I don't think so. The image suggestion work on Flink never progressed passed the original ticket.

Getting rid of service-scaffold-node also means we should get rid of servicelib-node since it was created for service-scaffold node (this is the reason why service-scaffold node depends on packages that don't exist. This project never finished)

In T344730#9848618, @Ottomata wrote:

Added jsonschema-tools to list, as it is similar to node-rdkafka-factory and eventgate.

Don't forget that any CI that has a production deployment pipeline needs the repo to be added to trusted runners and also have their tags protected (Slack thread on protecting tags)

In T344730#9838311, @Ottomata wrote:

Yes, that's actually what I do for service-utils

Very cool!

@tchin should we make that a reusable gitlab_ci template job in workflow_utils?

This might be harder than I thought. Creating a dummy google account to act as the receiver seems off the table. All of Google's APIs require OAuth or some manual way for the user to sign in. There is no way to make a pure bot account, and also no good way to automate login without being slapped by a ban.

In T344730#9838008, @Ottomata wrote:

Can we publish and depend on npm pacakges from gitlab, like we do for python wheels?

Sounds good to me. service-scaffold-node was started to turn service-template-node into a group of libraries and is basically superseded by my effort to replace service-runner (T360924) which is mostly completed

Would be nice to get a confirmation for archiving node-rdkafka-statsd since it'll progress T349118

In T347498#9791831, @Ottomata wrote:

@tchin, if we are able to get off of old service-runner, would your new framework take care of this?

Has this project been discussed across the WMF/Community?

It would be great if there was a RFC process, but there has at least been discussions about what to do with service-runner and this project is on the radar to the entirety of Data Platform Engineering and some people on the MW engineering team and the language team. It was also posted on slack on #engineering-all to give people a head's up just in case there was another team working on something similar. If there's one thing I'm sure about is that the consensus is that we need a replacement, whether or not this is it.

Config store repo does CI checks for jsonschema correctness and config values against its jsonschema. The Datasets Config service repo has dockerized CI using Kokkuri and Blubber.

If it's to a point where we even need to use a new name, might as well break everything. I'd love to join in on the fun

Looking at the logs, this seems to coincide with the redaction patch to eventstreams, but looking at the code I'm having a hard time finding where a memory leak could've happened... more confusing that it's just 1 or 2 pods hitting the limit

Using lz4 compression works but checking it with parquet-tools doesn't. I see something like compression: UNKNOWN (space_saved: -25%) Seems like a known issue.

INSERT OVERRIDE with PARTITION also doesn't work anymore because Iceberg uses hidden partitioning so had to enable Spark's dynamic overwrite
https://iceberg.apache.org/docs/latest/spark-writes/#insert-overwrite

TIL when setting the compression codec to snappy, Iceberg doesn't end the files in hdfs with .snappy.parquet. I had to check if the format was correct using parquet-tools.

Tested to see if the COALESCE hints still work in Iceberg by creating 2 tables and filling then with/without the hint. It still seems to work.

Tested on a stat machine with

CREATE EXTERNAL TABLE IF NOT EXISTS `aqs_hourly`(  
    `cache_status`      string     COMMENT 'Cache status',  
    `http_status`       string     COMMENT 'HTTP status of response',  
    `http_method`       string     COMMENT 'HTTP method of request',  
    `response_size`     bigint     COMMENT 'Response size',  
    `uri_host`          string     COMMENT 'Host of request',  
    `uri_path`          string     COMMENT 'Path of request',  
    `request_count`     bigint     COMMENT 'Number of requests',  
    `hour`              timestamp  COMMENT 'The aggregated hour. Covers from minute 00 to 59'  
)  
USING ICEBERG
PARTITIONED BY (days(hour))
;

And

spark3-sql --master yarn --executor-memory 8G --executor-cores 4 --driver-memory 2G --conf spark.dynamicAllocation.maxExecutors=64 \
-f aqs_hourly_iceberg.hql  \
-d source_table=wmf.webrequest \
-d webrequest_source=text \
-d destination_table=tchin.aqs_hourly \
-d coalesce_partitions=1 \
-d year=2023 \
-d month=12 \
-d day=3 \
-d hour=0

I think the per-image quota should probably be increased. I tested building a few projects locally and a project with NodeJS and 0 dependencies results in a built image that's 805.58 MB. One with only VueJS as a dependency bumps it up to 858.13 MB. I'm probably not going to be the last one who needs more than 200 MB of working space :/

Example error:

step-export: 2023-11-13T05:41:56.835942824Z ERROR: failed to export: failed to write image to the following tags: [tools-harbor.wmcloud.org/tool-dpe-alerts-dashboard/tool-dpe-alerts-dashboard:latest: PATCH https://tools-harbor.wmcloud.org/v2/tool-dpe-alerts-dashboard/tool-dpe-alerts-dashboard/blobs/uploads/b62dd944-4fad-4ee8-b900-8409f7860d6c?_state=REDACTED: unexpected status code 413 Request Entity Too Large: <html>
step-export: 2023-11-13T05:41:56.835973012Z <head><title>413 Request Entity Too Large</title></head>
step-export: 2023-11-13T05:41:56.835976984Z <body>
step-export: 2023-11-13T05:41:56.835979969Z <center><h1>413 Request Entity Too Large</h1></center>
step-export: 2023-11-13T05:41:56.835983468Z <hr><center>nginx/1.18.0</center>
step-export: 2023-11-13T05:41:56.836002364Z </body>
step-export: 2023-11-13T05:41:56.836005027Z </html>
step-export: 2023-11-13T05:41:56.836008032Z ]
step-export: 
step-results: 2023-11-13T05:41:57.433667715Z 2023/11/13 05:41:57 Skipping step because a previous step failed

Current version of the writeup is here

If we do introduce something, we should use JSDoc3 and follow what's happening on this ticket T138401

DeliveryGuarantee.AT_LEAST_ONCE: The sink will wait for all outstanding records in the Kafka buffers to be acknowledged by the Kafka producer on a checkpoint. No messages will be lost in case of any issue with the Kafka brokers but messages may be duplicated when Flink restarts because Flink reprocesses old input records.

https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/kafka/#fault-tolerance

Unaligned checkpoints didn't work. Maybe it's because of data being moved around to new brokers and Kafka is too overloaded.

@bking Gabriele is currently on sick leave but yes let's try incrementing the helm chart version

Associated GitHub PR: https://github.com/wikimedia/jsonschema-tools/pull/48

Seems like in jsonschema-tools the enums are only validated through ajv and their strict union type checking allows null so will have to implement the check ourselves

While adding a workaround to T344235, I noticed that additionalProperties isn't very well represented in DataHub.

"custom_data": {
    "additionalProperties": {
        "properties": {
            "data_type": {
                "type": "string",
                "enum": ["number", "string", "boolean", "null"],
            }
        }
    },
    "propertyNames": {
        "maxLength": 255,
        "minLength": 1,
        "pattern": "^[$a-z]+[a-z0-9_]*$",
    },
},

Just shows up in DataHub as a Struct with no defined nested fields (which I guess makes sense, but is not helpful).

From the recent meeting:

• Event Streams will be the name of the platform
• Streams are upstream to Kafka topics

After experimenting a lot, I have a Datahub transformer for Kafka that generates an Event Streams platform, adds description, schema, and path. However, I don't know if it should be a transformer since it's doing a bit more than just transforming.

Since Datahub has the concept of platforms, I think the best way forward is to have a separate platform called Event Streams where the datasets under it are the streams defined in the stream config. We can then keep the Kafka platform for all the individual Kafka topics. Then what we can do is have a transform attached to the current Kafka ingestion recipe that will attach the schemas to the individual topics when supported but also at the same time insert the streams into the Event Streams platform. This way we can have the schemas on both the stream and its topics

On the wiki for schema guidelines there's a blanket statement that all modifications should be backwards compatible - I assume this doesn't apply to major version changes so will note that

I could try taking a crack at it

In T337400#8907322, @dancy wrote:

Can you point me to the job output showing the hang?

Is there a benefit to doing this in blubber though?

The cookiecutter template also does this via a post-generation hook
https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/blob/main/cookiecutter-event-pipeline/hooks/post_gen_project.py

Ok so just recounting my experiments: