Osm Admin

I waited for this day too long. Today, I've tried out GitPod, and made it work for a project based on Osm Admin.

Today, I reiterated on what's left in this iteration, and finished the side menu view, and created a trivial home page for the admin area.

After getting the main branch all-green (tests pass, CLI and UI works as expected), I decided that from now on, I'll keep it always green, and use feature branches for all major development.

Then, I started implementing the main menu of the admin area.

Yesterday, I created a template for new projects powered by Osm Admin.

This way, you can create and publish a project in minutes, just follow the README. Yay!

Implemented in Osm Admin:

• new ALTER TABLE algorithm
• NULL to non-NULL data conversion
• new query test suite

This time, I tackled adding an explicit property to an existing table, and then making it non-nullable.

It raised the need for data conversions - additional handling of the existing data - or otherwise, the database engine fails, or the data becomes invalid.

And it made me split migrations in two parts - the one that runs before the data conversion, and the other one that runs afterwards.

While working on diff algorithm I noticed a certain pattern that I use over an over again.

Using this pattern, I implemented notification table diff.

Finally, I rearranged diff/migration code for better code readability.

Yesterday, I moved the table creation code into new schema diff migration engine.

Now, working on notification table migrations.

This time:

• I found out that the whole search index creation should be done during indexing, not migrations.
• I refactored diff algorithm and made it much easier to read.

Before digging into property diff algorithm, let's dig into how exactly it's used, and what other property-related logic is there.

Yesterday, I continued working on schema migrations:

• implemented table diff algorithm;
• took into consideration table renames.

After a system update, I've got some problem with my laptop, and one thing to check was system log files.

This got me thinking: what makes a good log for my own application? And I think that application logs are as useful as they are able to answer potential questions I may have when an application is in production.

Let's think what kind of questions I might have, and what information might be useful to answer them.

Recently, I fixed numerous issues to make the first migration tests work.

Then, I started implementing the schema diff algorithm - the basis for schema migrations.

More dots got connected in the schema migration design, see this post for details.

Then, I tried to return to TDDing it, but strange errors met me there, so I left them to be resolved in the next piece.

Schema migrations are going to be generated and executed automatically under gulp watch. It will result in lots of tiny migrations: add a column, change its type, make it not nullable, and so on.

It's convenient during development, but it might be not optimal to reapply in production. Indeed, it's better to convert a large table to new structure all at once rather than running a conversion for each of the tiny migration.

This observation contradicts the previously stated migration workflow, so let's update it.

In spirit of TDD, I started with a meaningful, but failing test that is reasonably fast. Then, I worked to make it green:

• implemented the loading of schema fixtures from the codebase;
• worked on generating migration files.

After drafting some ideas of how the schema migrations should actually work, let's start implementing it in TDD way.

The goal of the current iteration is to adjust the database schema and preserve/convert existing data according to any changes in schema classes, grids, forms or indexers.

This post presents some thoughts how it might work with class definitions.

After finishing the iteration "#17 Minimum UI", I revisited the goals and the scope of the minimum viable product, and picked the most pressing task for the next iteration - "#18 Database Schema Changes".

I implemented the last user action that is typical to a CRUD application - object deletion.

Done:

• routes responsible for creating new products, and modifying existing products
• incremental indexing
• cloning query objects
• INSERT INTO ... SELECT ... statements

After 3 days of intensive work, I've got dot syntax working!

Along the way, I refactored the query internals once again, which reminds me a heuristics saying that you've got to implement something three times to get it right :)

New query implementation uses the concept of formulas - SQL-like expressions used for selecting, filtering and sorting data. Currently, formulas are quite limited, but with time, they should be a really powerful feature.

I decided to support selecting, filtering and ordering by properties of a related object using dot syntax: product.price, parent.level. This syntax will automatically join the table that stored related objects.

I realized that implementing joins requires a better query model than my current naive wrapper around Laravel query. This article describes the new query model.

This article is about running SELECT and INSERT queries on data objects stored in the database.

In short, use the query() function. It runs on top of Laravel Query\Builder. It handles the mapping of data class properties onto table columns, and initiates the computation of indexed properties.

Currently, you can only SELECT and INSERT, other operations are coming soon.

Today, I implemented computation of indexed properties during insert. The implementation dictated some changes in indexed property definitions compare to the initial design, and how indexers are reflected into the data schema.

Also, I implemented a table query class working on the top of Laravel Query\Builder. More about it next time, for now I'll review how indexed properties are computed during INSERT operation.

It's often needed to compute, or index, data in database tables based on data in other tables.

This article is the first sketch of a consistent indexing solution.

I've finished refactoring how Osm Admin stores data objects in the database. Also, in order to support multi-website, multi-vendor, multi-language applications, I've introduced the concept of scopes.

Most reasoning from the first version is still valid, so let's take a fresh look of what's changed.

I developed dehydrate()/hydrate() functions for a very practical need - storing the data class schema:

• in cache - to reduce processing time spent on reflection;
• and in database - to compare current and new database schema, and only add new tables and columns.

This article describes the information stored in the schema, and unit tests that I prepared in order to be sure that schema classes are property hydrated.

As described in Migrations, most data objects of the same class will be stored in a database table.

But what about subclasses? In an e-commerce application, bags, dresses, and digital products, collectively known as subclasses, are all products stored in products table, and they may have bad-specific, dress-specific or digital product specific properties that should also be stored there.

This article describes how subclasses are defined and stored in Osm Admin.

Different data classes have repeating structural patterns. For example, most data classes stored in database tables have the auto-incremented id property. Use PHP traits to effectively introduce the same properties to different data classes over and over again.

I partly implemented data schema migration. It takes data class definitions, and incrementally creates or alters underlying database tables.

The article below describes how schema migration works, and what's not implemented yet, but most probably will be.

Note. This topic is continued in the new article.