Yeah I often see devs come up with complex architectures to just work around Lambda’s limitations. Things to keep them warm (but then fail because they don’t account for concurrent requests still hitting cold starts), multiple levels of Lambda functions to work around 15 minute time outs, and more. Just use the right tool for the job and look at Fargate or Batch.
It all depends on how it’s represented on disk though and how the query is executed. Sqlite only supports numbers and strings, and if you keep using a VARCHAR
, a read of those rows are going to have materialize a string into memory inside the sqlite library. DuckDB has more types, but if you’re using varchars everywhere, something has to read that string into memory unless you can push down logic into a query that doesn’t actually have to read the actual value, such as one that can use indices.
The best way is to change the representation on disk, such as converting low-cardinality columns like the station
into a numeric id. A standard int
being four bytes is a lot more efficient than an n-byte string + a header and it can be compared by value.
This is where file formats, like Parquet, shine. They’re oriented more towards parsing by systems. JSON is geared towards human parsing.
Not all filtering is the same. Client side filtering requires more data to passed over the network that then just gets dropped. It also means rules that are not shared across devices.
Most importantly, these use CSS filters which are computationally more expensive because it has to take an entire DOM element, serialize it to text, string search it vs a server side filter that can just look at a one or two field variables. Even if it’s not filtered in SQL on Lemmy’s side I’d say it’s still more efficient overall.
You do what you want, but adding extra work on the client side is not what I’d want for my users. Of course, if your Lemmy instance does not supporting filtering, then this is moot.
This is a good idea. It’s a lot easier to incentivize a maintainer who is already familiar with a project and invested in it with some money than it is to get a person who is unfamiliar with a project.
How much you should donate and how likely they are to agree depends on how complex your request is, whether they feel it fits in with the project for other people, and how busy they are.
I just saw this one mention endurain, a fitness tracker. I’ve been looking for something to self host data about my health, fitness, etc. Has anyone tried this or anything else in the self-hosted or open source fitness space?
Its not difficult for technical people like you or me, but my friend who just wants to watch their favorite show on my Plex on their TV won’t know how to traffic engineer the traffic over a Tailscale network to my network. My mom won’t be installing Tailscale on her laptop and phone.