Summary
[!note] 📚 Smart Assets Manager Series
- Why Storage Abstraction Matters ← you are here
- Four Backends, One Interface — May 18
- The Unified API: Credits and Rate Limiting — April 27
- Testing Strategy: Unit vs E2E — April 20
- 5 Edge Cases That Break Image APIs — June 1
- API Documentation: Swagger + Postman — March 30
When I started building Smart Assets Manager — a system for generating blog featured images, app store icons, and social media graphics in bulk — the storage question felt simple. Generate the image, upload it to Cloudinary, return the URL. Done.
That lasted about two weeks.
The third person to use the API came back with a question: “Can I run this locally during development without needing Cloudinary credentials?” Good question. The answer, with a hardcoded Cloudinary backend, was no.
A week later, a CI/CD pipeline integration request came in: the API was being called mid-pipeline to generate Open Graph images before deployment. Uploading to Cloudinary in that context created an external dependency that could fail independently of the deployment itself. The pipeline author wanted to receive the image bytes directly in the response and store them however they wanted.
Two weeks after that, an enterprise use case came up: a client whose data residency policy prohibited images from leaving their infrastructure. Cloudinary, as a third-party CDN, was off the table entirely.
Three different problems. Three different requirements. All pointing at the same root cause: the generator and the storage were coupled.
Takeaway: A storage backend hardcoded into an image generation API isn’t a storage decision — it’s a constraint that propagates through every use case you haven’t thought of yet. The question isn’t whether you’ll need flexibility; it’s whether you’ll have built it by then.
What Coupling Costs
The code at the time looked approximately like this — generator produces bytes, Cloudinary call produces URL, URL returned:
def generate_and_store(template_data: dict, width: int, height: int) -> str:
image_bytes = renderer.render(template_data, width, height)
# Upload directly to Cloudinary — no abstraction
result = cloudinary.uploader.upload(image_bytes, public_id=template_data["name"])
return result["secure_url"]
Clean, simple, works perfectly — until any of the three use cases above appears. At that point, this function has to be modified. Every test that mocks cloudinary.uploader.upload has to be updated. Every endpoint that calls this function inherits Cloudinary as a dependency. Every CI environment running tests needs Cloudinary credentials configured.
The change needed wasn’t to swap Cloudinary for something else. It was to move the “where does the image go” decision out of the generation code and into a parameter the caller controls. The generator should produce bytes. What happens to those bytes should be configured separately.
Four Use Cases, Four Backends
The three requests described above, plus production hosting, gave the shape of the abstraction:
| Backend | Caller need | What storage does |
|---|---|---|
| Cloudinary | Production hosting, shareable URLs | Upload to CDN, return public URL |
| Local file system | Development, on-premise requirements | Write to disk, return localhost URL |
| Amazon S3 | Production workloads on AWS infrastructure | Upload to bucket, return S3 URL |
| Direct download | CI/CD pipelines, API consumers | Skip storage, return base64 data URI in response |
The caller specifies which backend to use via a storage parameter in the API request. The generator doesn’t change — only the backend dispatched by the factory changes.
The direct download backend deserves a note: it was the use case nobody had planned for and turned out to be the most frequently used by developers. Instead of uploading anywhere, it base64-encodes the raw image bytes and returns them inline in the API response:
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...
No credentials required. No storage costs. No round-trip latency. For development, testing, and automated pipelines where the image will be processed by the downstream system anyway, this is the right default — and it only exists because the abstraction made adding it cost a few hours rather than a refactor.
[!note] The backends you don’t plan for The direct download backend — returning base64 in the response — was the most useful backend nobody thought to request upfront. Storage abstractions justify themselves partly by how cheaply they let you add the backends you hadn’t anticipated.
What the Abstraction Actually Costs
The pattern that enables all of this is not complex:
- An abstract base class with two required methods —
upload()returns a URL,generate_signed_url()returns a time-limited URL for private assets - Four concrete implementations — one class per backend, each implementing the same two methods
- A factory function — takes the
storageparameter from the request, returns the appropriate backend instance
That’s a single file, approximately 250 lines across the base class and four implementations. The investment is bounded and front-loaded. The benefit — the ability to add a new backend, swap the production backend, or test without external credentials — compounds with every feature added to the system after it’s in place.
Retrofitting this abstraction after the system had grown would have required touching the generator, every API endpoint, every test, and every deployment configuration. The cost of doing it upfront is always lower than the cost of doing it later. But the comparison only becomes visible in hindsight, which is what makes “just use Cloudinary” feel like a reasonable choice at the start.
What’s Next
The architecture case is made. The next post covers the actual implementation: how four storage backends share one interface, including the privacy controls, the cleanup-on-failure logic, and the byte-signature detection that makes format inference reliable.
