In partnership with

Hello “👋

Welcome to another week, another opportunity to become a Great Backend Engineer.

Today’s issue is brought to you by Masteringbackend → A great resource for backend engineers. We offer next-level backend engineering training and exclusive resources.

Before we get started, I have a few announcements:

I have a special gift for you: You will love this one.

The ultimate “Land Your Dream Job” Challenge is here.

We are launching the ultimate guide to land your dream job in any programming language you choose. We are starting with the Python Programming language.

Land your dream Python Job in 90 days by shipping 30 Python projects in 30 days by completing our daily tasks.

It’s a cohort-based and project-focused challenge where you will be challenged to build 30 Python projects in 30 days.

Here are what you will get:

• Ship 30+ Python backend projects in 30 days.

• Instant Access to all 30+ videos

• Access to data structure and algorithm interview kits.

• Access our Complete Backend Job Preparation kits (Resume, Cover letter reviews, mock interviews, and job placements).

• Join & learn from a thriving community of helpful students & alumni from top companies.

Limited Access. First 400 students at $54 others at $100 (We have only 120 slots left.)

Click here to reserve your spot.

There’s a reason 400,000 professionals read this daily.

Join The AI Report, trusted by 400,000+ professionals at Google, Microsoft, and OpenAI. Get daily insights, tools, and strategies to master practical AI skills that drive results.

Sign up now for free and work smarter, not harder.

In our previous issue on Building JSON/RESTful APIs, we explored the principles of designing scalable, maintainable, and efficient APIs using REST and JSON.

However, as your API scales and begins handling large datasets, it becomes impractical to return all data in a single response.

This is where pagination comes into play.

In this issue, we will explore the role of pagination in API design, compare different pagination strategies, and provide detailed implementation examples.

By the end, you'll have a solid understanding of how to implement pagination effectively in your JSON/RESTful APIs.

Why Pagination Matters in API Design

APIs often need to return large datasets, such as:

• A list of users in a system

• Products in an e-commerce catalog

• Log entries in a monitoring system

If an API were to return thousands (or millions) of records in a single response, the consequences would be severe:

• Slow Performance – Large responses increase response time and slow down both the server and the client.

• High Memory Consumption – Loading all records into memory at once puts unnecessary strain on server resources.

• Network Overload – Transferring huge JSON responses can cause bandwidth congestion and increase API latency.

Pagination solves these problems by breaking the dataset into smaller chunks (pages), allowing clients to request only what they need.

Pagination Strategies

There are several ways to implement pagination, each with its trade-offs:

1. Limit-Offset Pagination (Basic Pagination)

2. Cursor-Based Pagination

3. Time-Based Pagination

4. Keyset Pagination

Let's break them down in detail.

Limit-Offset Pagination (Traditional Approach)

How it Works:

• Clients specify two query parameters:

  • limit (number of records per page)

  • offset (starting position of the data)

• The server retrieves records starting from the given offset up to the limit.

Example API Call:

GET /users?limit=10&offset=20

SQL Query Example:

SELECT * FROM users ORDER BY id LIMIT 10 OFFSET 20;

Implementation in Node.js (Express + Prisma ORM)

app.get('/users', async (req, res) => {
    const limit = parseInt(req.query.limit) || 10;
    const offset = parseInt(req.query.offset) || 0;

    const users = await prisma.user.findMany({
        skip: offset,
        take: limit,
        orderBy: { id: 'asc' },
    });

    res.json({ users, limit, offset });
});

Pros:

• Simple to implement and understand

• Easy to integrate with SQL databases

Cons:

• Performance issues with large datasets (OFFSET grows linearly)

• Skipping rows is inefficient (especially with big tables)

Cursor-Based Pagination

This strategy is great for large datasets and a very efficient pagination strategy for millions and billions of datasets.

How it Works:

• Instead of specifying an offset, the client provides a cursor (a unique identifier for the last seen record).

• The API retrieves the next N records starting after the cursor.

Example API Call:

GET /users?limit=10&cursor=100

SQL Query Example:

SELECT * FROM users WHERE id > 100 ORDER BY id ASC LIMIT 10;

This query will retrieve all the items where the ID is greater than 100 and will stop at the 10th element because of the LIMIT keyword. Then next iteration will start from 111 like that till it finishes the list.

Implementation in Node.js (Express + Prisma)

app.get('/users', async (req, res) => {
    const limit = parseInt(req.query.limit) || 10;
    const cursor = req.query.cursor ? parseInt(req.query.cursor) : null;

    const users = await prisma.user.findMany({
        where: cursor ? { id: { gt: cursor } } : {},
        take: limit,
        orderBy: { id: 'asc' },
    });

    // Set next cursor (last item ID)
    const nextCursor = users.length ? users[users.length - 1].id : null;

    res.json({ users, nextCursor });
});

Pros:

• Efficient for large datasets (avoids expensive OFFSET operations)

• Fast and scalable

Cons:

• Requires a unique and indexed sorting field (like id)

• Cannot jump to arbitrary pages easily

Time-Based Pagination

Time-based pagination is mostly used for real-time pagination, and it’s very efficient when used with WebSocket and other real-time applications.

How it Works:

• Instead of an offset, pagination is based on timestamps.

• Clients fetch records newer or older than a given timestamp.

Example API Call:

GET /logs?limit=10&after=2024-02-17T12:30:00Z

SQL Query Example:

SELECT * FROM logs WHERE created_at > '2024-02-17T12:30:00Z' ORDER BY created_at ASC LIMIT 10;

Implementation in Node.js (Express + Prisma)

app.get('/logs', async (req, res) => {
    const limit = parseInt(req.query.limit) || 10;
    const after = req.query.after ? new Date(req.query.after) : null;

    const logs = await prisma.logs.findMany({
        where: after ? { createdAt: { gt: after } } : {},
        take: limit,
        orderBy: { createdAt: 'asc' },
    });

    const nextAfter = logs.length ? logs[logs.length - 1].createdAt : null;

    res.json({ logs, nextAfter });
});

Pros:

• Ideal for real-time updates (chat apps, logs, notifications)

• Efficient indexing with timestamp columns

Cons:

• Prone to issues with time zone inconsistencies

• Does not support jumping to specific pages

Keyset Pagination

Keyset-based pagination is an alternative cursor-based pagination.

How it Works:

• Uses multiple unique columns (e.g., id + created_at) to determine the next page.

• It prevents inconsistencies when sorting by fields that are not unique.

Example Query:

SELECT * FROM orders 
WHERE (created_at, id) > ('2024-02-17T12:30:00Z', 100) 
ORDER BY created_at ASC, id ASC LIMIT 10;

Pros:

• More stable sorting for non-unique fields

• Efficient for ordered datasets

Cons:

• More complex implementation

Best Practices for API Pagination

1. Choose the Right Pagination Strategy

   • Use limit-offset for small datasets.

   • Use cursor-based for large datasets.

   • Use time-based for real-time applications.

2. Return Pagination Metadata

   • Always include pagination metadata (nextCursor, nextPage, totalPages).

3. Optimize Database Queries

   • Use indexed fields for filtering (e.g., id, created_at).

   • Avoid OFFSET in large tables when possible.

4. Consider Client Usability

   • Ensure clients can navigate easily (next, previous, first, last links).

Pagination is an essential component of API design, ensuring efficient data retrieval while maintaining performance and scalability. Choosing the right pagination strategy depends on your API's requirements—limit-offset is simple but inefficient for large datasets, while cursor-based and keyset pagination is ideal for high-performance applications.

Implementing these best practices, backend engineers can design scalable, performant APIs that provide clients with a seamless experience while optimizing backend resources.

Did you learn any new things from this newsletter this week? Please reply to this email and let me know. Feedback like this encourages me to keep going.

Remember to start learning backend engineering from our courses:

Get a 50% discount on any of these courses. Reach out to me (Reply to this mail)

1. Become a Node.js Backend Engineer is Live

2. Become a Rust Backend Engineer is Live

3. Become a Python Backend Engineer is Live

4. Become a Java + Spring Backend Engineer is Live

Top 5 Remote Backend Jobs this week

Here are the top 5 Backend Jobs you can apply to now.

👨‍💻 FaceUp
✍️ Node.js Backend Developer for FaceUp
📍Remote, Prague, Brno
💰 Click on Apply for salary details
Click here to Apply for this role.

👨‍💻 ROIVENUE
✍️ Full-stack Software Developer
📍Remote, Prague
💰 Click on Apply for salary details
Click here to Apply for this role.

👨‍💻 Homepage
✍️ Backend Developer PHP
📍Remote, Worldwide
💰 Click on Apply for salary details
Click here to Apply for this role.

Want more Remote Backend Jobs? Visit GetBackendJobs.com

Backend Engineering Resources

1. Backend Engineering Hub

2. All Backend Books

3. Visit our Backend Store

4. Join our Community

5. Backend Engineering Courses

LAST WORD 👋 

How am I doing?

I love hearing from readers, and I'm always looking for feedback. How am I doing with The Backend Weekly? Is there anything you'd like to see more or less of? Which aspects of the newsletter do you enjoy the most?

Hit reply and say hello - I'd love to hear from you!

Stay awesome,
Solomon

I moved my newsletter from Substack to Beehiiv, and it's been an amazing journey. Start yours here.