Hello everyone!
For the last week or so, I’ve been improving the backend of Solar Marine notifications and the insight hub. (I wrote more about it at the end of this post if you’re interested in technical details).
These changes will allow me to add new things to the hub, and I have a few ideas already:
Idea #1: Reports / summaries
Weekly (and possibly monthly) summary page with an overview of everything that happened in the store, such as:
- New products, restocked products, sold-out products
- Totals for # of products available, # of products sold etc., possibly by categories, such as product type (dolls/part/apparel etc), skintone, sculpt type (anime, semireal), body type (classic, pear) etc.
- Summary of all other changes, such as price changes, updates of photos or descriptions of the dolls etc.
Idea #2: Cocoa canary
We all know what happened to Milk. Cocoa “canary” would be a specialized analysis looking at trends of Cocoa skintone product in comparison to other skintone over time. Like a canary in a coal mine it would help notice signs of Cocoa becoming discontinued.
We’d examine the dynamics of products available/unavailable, stock quantities, sales, restocks, etc.
Idea #3: Running-out-soon products
Based on the current stock availability and the pace of sales in the preceding period, it would list the products that are expected to run out soon.
It would be a simple and probably rather inaccurate model at first, but with more data available, it should be possible to get it to a fairly decent accuracy.
It could also be combined at a later stage with the history of restocks for any product for additional insight into potential “next chance” after it sells out.
As usual, I will make a poll on X for these ideas, and if you have any of your own, please let me know in the comments, replies or DMs.
Backend design issue explained
For nerds.
Short explanation
The notification bot processes a lot of data and sends relevant notifications but currently doesn’t save 95% of that data for later use. I need to make changes so it can be reused on the insights hub. The backend will use database to store all important info, so it can be accessed and reused later for different notifications or reports.
Long explanation
The notification bot currently downloads product data from the website and immediately compares it with the previous snapshot. It looks for all differences and notifies about relevant ones (currently only availability) on notification channels. All other differences are detected but not stored. Basically, I see them on the terminal.
This design has two issues (if not more):
- As already mentioned, all effort put into detecting changes in the store (other than product availability) is lost.
- It’s rather difficult to decouple the process of detecting changes from the process of sending notifications. If one part goes wrong, everything goes wrong. Notifications can’t be really delayed, queued, or grouped.
So, to fix this, I’m redoing the backend to introduce a “step” between data processing and using that processed data.
The most obvious solution is, of course, to use a database where all info is stored, alongside the change log.
In such a setup, the change log is basically for free because database triggers take care of detecting changes whenever there’s an update.
All changes stored in the database will be easily accessible to the notification service and insights hub at any time. This allows for new types of notifications about various changes that could be delayed or grouped. For example, summaries about all items restocked and/or sold out this week or changes to prices or other attributes.
It will also be possible to browse the history of all changes and filter them by type, product, dates, etc. This functionality will come to the Solar Marine Hub in the future.