Object Storage Explained: How Object Storage Works and When to Use It

When you upload a photo to a cloud app, stream a video from a service like Netflix, or back up a database, the bytes are almost certainly ending up in object storage. Object storage has become the default architecture for cloud data, but the name obscures how different it is from the file system on your laptop or the disk attached to a server. This guide explains what makes object storage different, how services like Amazon S3 implement it, and when to choose it over block or file storage.

What is object storage

Object storage (also called object-based storage or blob storage) is a data storage architecture that manages data as "blobs" or "objects", as opposed to other storage architectures like file systems, which manage data as a file hierarchy, and block storage, which manages data as blocks within sectors and tracks.

Each object is typically associated with a variable amount of metadata, and a globally unique identifier. Object storage can be implemented at multiple levels, including the device level (object-storage device), the system level, and the interface level. In each case, object storage seeks to enable capabilities not addressed by other storage architectures, like interfaces that are directly programmable by the application, a namespace that can span multiple instances of physical hardware, and data-management functions like data replication and data distribution at object-level granularity.

The term blob was coined by Jim Starkey while working at Digital Equipment Corporation, and was later adopted for the Rdb/VMS database. The humorous backronym "binary large object" was retroactively applied to it. The architectural research that shaped modern object storage was led by Garth Gibson at Carnegie Mellon in 1995, in a project called Network-Attached Secure Disks (NASD), which promoted the idea of splitting namespace operations from bulk reads and writes to optimize scale. Gibson later co-founded Panasas to commercialize these ideas, and one of his collaborators, Howard Gobioff, went on to co-design the Google File System.

How objects differ from files and blocks

A traditional file system organises data in a tree of folders and files. Each file has a path, a fixed set of metadata (name, size, timestamps, permissions), and a set of bytes. To find a file, you walk the tree. A block storage system breaks data into fixed-size blocks, each addressed by a logical block number, and reassembles them on read. Block storage is what an operating system mounts as a local disk.

An object is a different kind of unit. Instead of a path, each object has a unique identifier (often a URL-like key) and a rich, application-defined set of metadata attributes. The container that holds objects is usually called a bucket (in S3) or a container (in Azure Blob Storage). Within a bucket there is no folder hierarchy: all objects live in a flat namespace, and the application gives each object a key. The combination of a key, a value (the bytes), and a set of attributes is what object storage calls an "object".

Object storage also allows the addressing and identification of individual objects by more than just file name and file path. Object storage adds a unique identifier within a bucket, or across the entire system, to support much larger namespaces and eliminate name collisions.

Rich metadata and custom attributes

File systems have a small, fixed set of metadata: filename, creation date, type, owner. Object storage explicitly separates file metadata from data to support additional capabilities. Object storage provides for full function, custom, object-level metadata, which can be used to:

• capture application-specific or user-specific information for better indexing
• support data-management policies (for example, a policy to drive object movement from one storage tier to another)
• centralise management across many nodes and clusters
• optimise metadata storage (encapsulated, database, or key-value) and caching/indexing independently from the data storage

This is the part that makes object storage useful for unstructured data: you can tag a video with its encoding, duration, and content rating; you can tag a backup with the database it came from and the time of the snapshot; and the storage system can use those tags to drive lifecycle policy without the application having to re-read the object.

Programmatic access through REST APIs

Object storage provides programmatic interfaces to allow applications to manipulate data. At the base level, this includes create, read, update, and delete (CRUD) functions for basic read, write, and delete operations. Some object storage implementations go further, supporting object versioning, object replication, life-cycle management, and movement of objects between different tiers and types of storage. Most API implementations are REST-based, allowing the use of many standard HTTP calls.

That is why the same HTTP verbs that power the web (GET, PUT, POST, DELETE) are also the operations you use to read, write, and delete objects. From an application's point of view, an object store is just an HTTP service.

How Amazon S3 implements object storage

Amazon S3 is the service that made object storage the default. Launched in the United States on March 14, 2006, and in Europe in November 2007, S3 is built on the same scalable storage infrastructure that runs Amazon's e-commerce network.

The basic storage units of Amazon S3 are objects which are organised into buckets. Each object is identified by a unique, user-assigned key. Buckets can be managed using the console, programmatically with the AWS SDK, or the REST API. Objects can be up to five terabytes in size. If an object is larger than 5 GB, it must be divided into chunks prior to uploading, and any object larger than 5 GB must be uploaded via the S3 multipart upload API.

Requests are authorised using an access control list associated with each object bucket, and S3 supports versioning, which is disabled by default. The AWS authentication mechanism also allows the creation of authenticated URLs valid for a specified amount of time, so a private object can be shared temporarily without making the bucket public.

S3 offers nine different storage classes with different levels of durability, availability, and performance requirements. The default is S3 Standard, for frequently accessed data. Other classes target infrequent access, archival, and lower-cost long-term retention. S3 can also be used to replace static web-hosting infrastructure: an S3 bucket can be configured with an index document and an error document, and every object can be served as a public HTTP URL. As of 2025, S3 stores 500 trillion objects across hundreds of exabytes and serves 200 million requests per second, peaking at about one petabyte per second in bandwidth.

Object storage vs key-value storage

The border between an object store and a key-value store is blurred, with key-value stores being sometimes loosely referred to as object stores. They are similar in two respects: the object identifier (the equivalent of a key) can be an arbitrary string, and the data can be of an arbitrary size.

The key differences are these. Object stores allow one to associate a limited set of attributes (metadata) with each piece of data; the combination of a key, value, and set of attributes is the object. Object stores are optimised for large amounts of data (hundreds of megabytes or even gigabytes), whereas key-value stores are typically used for relatively small values (kilobytes). Finally, object stores usually offer weaker consistency guarantees, such as eventual consistency, whereas key-value stores often offer strong consistency. This is the trade-off that lets object stores scale to trillions of objects across many regions.

The S3-compatible ecosystem

The broad adoption of Amazon S3 and related tooling has given rise to competing services based on the S3 API. These services use the standard programming interface but are differentiated by their underlying technologies and business models. A standard interface enables better competition from rival providers and allows economies of scale in implementation.

Several storage providers such as Cloudian, Backblaze B2, and Wasabi offer S3-compatible storage with options of on-premises and private cloud deployments. Several open-source implementations of S3-compatible object-store servers can be self-hosted, such as Garage. IBM Cloud Object Storage also offers a cloud-based S3-compatible option. Cloud providers like Google Cloud Storage (May 2010) and Microsoft Azure Blob Storage have object products of their own; the S3 API has effectively become the de facto interface that most clients are written against.

Where object storage is used

Object storage systems allow retention of massive amounts of unstructured data in which data is written once and read once (or many times). Object storage is used for purposes such as storing videos and photos, songs on streaming services, and files in online collaboration services. The vast majority of cloud storage available in the market leverages an object-storage architecture.

Concretely, object storage is the right choice when the workload is dominated by large, write-once-read-many (or read-many-times) files: media assets, backups, log archives, software binaries, data lakes, and the static files behind a web or mobile application. Cloud functions read configuration and templates from object storage; CDNs source content from object storage; data lake analytics query Parquet files in object storage; and machine learning training pipelines read datasets from object storage.

When object storage is the wrong choice

One of the limitations with object storage is that it is not intended for transactional data, as object storage was not designed to replace NAS file access and sharing; it does not support the locking and sharing mechanisms needed to maintain a single, accurately updated version of a file. Editing a single byte in an object means re-uploading the whole object, and the object does not provide the random-access read/write pattern a database or a block device needs.

If your application needs random writes, low-latency reads of small records, or a shared filesystem with locking, you want a block device (for example, an EBS volume), a relational database, or a network file system, not an object store.

FAQ

What is the difference between object storage and a file system?

A file system organises data in a hierarchy of folders and files, with a fixed set of metadata per file. Object storage organises data as flat objects in buckets; each object has a unique key, the bytes, and a rich set of application-defined metadata attributes, and there are no folders.

Is Amazon S3 the same as object storage?

Amazon S3 is one specific implementation of object storage, and the one that made the architecture mainstream. Many other services (Google Cloud Storage, Azure Blob Storage, Backblaze B2, Wasabi, Cloudflare R2, and self-hosted options like MinIO) implement the same general model and often expose the same S3 API.

Can I modify part of an object?

No. Object storage treats each object as an opaque blob. To change any byte, the application has to upload a new object (typically with the same key) and replace the old one. This is the price of the architecture's simplicity and scale; it is also why databases, which need in-place updates, are built on top of block storage rather than object storage.

How large can an object be?

In Amazon S3, an object can be between 0 bytes and 5 TB. Objects larger than 5 GB must be uploaded using the multipart upload API. Other object services have their own limits, but the order of magnitude is the same.

How is object storage different from block storage?

Block storage breaks data into fixed-size blocks addressed by number and reassembles them on read; it is what an operating system mounts as a local disk and is the right choice for databases, boot volumes, and any workload that needs random read/write access. Object storage treats each file as an opaque object addressed by key; it is cheaper per gigabyte at scale, but you cannot edit part of an object in place.

Sources

• Wikipedia — Object storage (en.wikipedia.org/wiki/Object_storage, CC BY-SA 4.0)
• Wikipedia — Amazon S3 (en.wikipedia.org/wiki/Amazon_S3, CC BY-SA 4.0)