summaryrefslogtreecommitdiff
path: root/sfeed_update
AgeCommit message (Collapse)Author
2021-05-27sfeed_update: fix message when the configuration file does not existHiltjo Posthuma
When sfeed_update was called without using a parameter and it used the default and this path did not exist it would incorrectly print: Configuration file "" does not exist or is not readable. See sfeedrc.example for an example. Make the error message a bit shorter too. This was a partial regression of commit df74ba274c4ea5d9b7388c33500ba601ed0c991d
2021-03-03sfeed_update: return instead of exit in main() on successHiltjo Posthuma
This is useful so the script can be included, call main and then have additional post-main functionality.
2021-03-01sfeed_update: fix baseurl substitutionHiltjo Posthuma
Follow-up from a rushed commit: commit 58555779d123be68c0acf9ea898931d656ec6d63 Author: Hiltjo Posthuma <hiltjo@codemadness.org> Date: Sun Feb 28 13:33:21 2021 +0100 sfeed_update: simplify, use feedurl directly This also make it possible to use non-authoritive URLs as a baseurl, like "magnet:" URLs.
2021-03-01sfeed_update: simplify, use feedurl directlyHiltjo Posthuma
This also make it possible to use non-authoritive URLs as a baseurl, like "magnet:" URLs.
2021-02-05sfeed_update: $SFEED_UPDATE_INCLUDE: be a bit more precise/pedanticHiltjo Posthuma
2021-01-27sfeed_update: $SFEED_UPDATE_INCLUDE: be a bit more precise/pedanticHiltjo Posthuma
2021-01-27sfeed_update: allow to reuse the code more easily as an included scriptHiltjo Posthuma
This adds a main() function. When the environment variable $SFEED_UPDATE_INCLUDE is set then it will not execute the main handler. The other functions are included and can be reused. This is also useful for unit-testing.
2021-01-27sfeed_update: separate code of parallel exection and feed() into a _feed() ↵Hiltjo Posthuma
handler This is useful to be able to reuse the code (together with using sfeed_update as an included script, coming in the next commit).
2021-01-27sfeed_update: shuffle code getting the path of the feedurl to make the ↵Hiltjo Posthuma
basesiteurl Move it closer before it is used.
2021-01-27sfeed_update: change parse failure error messageHiltjo Posthuma
"(FAIL CONVERT)" -> "(FAIL PARSE)". Convert may be too similar to text encoding conversion.
2021-01-27sfeed_update: add an overridable parse() function, using sfeed(1) by defaultHiltjo Posthuma
This can be useful to make more cleanly make connector scripts. This does not necesarily even have to be in the sfeed(5) format.
2021-01-24sfeed_update: print the filename again as passed as a parameterHiltjo Posthuma
... and do not show stderr of readlink.
2021-01-16sfeed_update: typo in commentHiltjo Posthuma
2021-01-16sfeed_update: improve consistency of feed creation and mergingHiltjo Posthuma
- Improve feed creation with empty results and new feed files. Always make sure the file is created even when it is new and there are also no items (after filtering). - Consistency: always use the same feed file for merging. Do not use "/dev/null" when it is a new file. This works using sort, but is ugly when the merge() function is overridden and does something else. It should be the feed file always.
2021-01-16sfeed_update: make convertencoding() consistent with other overridable functionsHiltjo Posthuma
This adds the name as the first parameter for the convertencoding() function, like filter, merge, order, etc. This can be useful to make an exception rule for text decoding in a more clean way.
2021-01-01sfeed_update: if baseurl is empty then use the path from the feed by defaultHiltjo Posthuma
Feeds should contain absolute urls, but if it does not have it then this makes it more convenient to configure such feeds.
2020-02-23sfeed_update: don't preserve permissions of tmp files by moving, so copyHiltjo Posthuma
noticed on DragonFlyBSD where it prints a warning when moving the file from /tmp. To reproduce it: touch /tmp/file mv /tmp/file ~/ On other systems this would not print a warning, but it would preserve the group permissions etc.
2020-02-04sfeed_update: tiny style fix in commentHiltjo Posthuma
2019-06-11sfeed_update: clarify a comment about being non-optimalHiltjo Posthuma
2019-05-15README and sfeed_update: use names (order vs sort) and in the execution orderHiltjo Posthuma
2019-05-12sfeed_update: cleanup proper encoding file (if any)Hiltjo Posthuma
2019-05-02sfeed_update: disable If-Modified-Since by defaultHiltjo Posthuma
2019-04-14sfeed_update: fix typo in commentHiltjo Posthuma
2019-04-14sfeed_update: rename fetchfeed to fetchHiltjo Posthuma
... and simplify example in README.
2019-04-14sfeed_update improvementsHiltjo Posthuma
- Better checking and verbose logging (on failure) of each stage: fetchfeed, filter, merge, order, convertencoding. This makes sure on out-of-memory, disk-space or other resource limits the output is not corrupted. - This also has the added advantage it runs less processes (piped) at the same time. - Clear previous unneeded file to preserve space in /tmp (/tmp is often mounted as mfs/tmpfs). - Add logging function (able to override), use more logical logging format (pun intended). - Code-style: order overridable functions in execution order.
2019-03-01sfeed_update: remove wrong/incomplete commentHiltjo Posthuma
2018-10-24sfeed_update: fix wrong comment "temporary file" -> "temporary directory"Hiltjo Posthuma
2018-10-24Revert "sfeed_update: replace non-POSIX mktemp with $$"Hiltjo Posthuma
This reverts commit 8699fa2bb4c75670952fee503a58ca4a652627eb. There is a regression in directory permissions among other things.
2018-10-11sfeed_update: replace non-POSIX mktemp with $$Hiltjo Posthuma
+ fix wrong comment "temporary file" -> "temporary directory".
2018-10-06sfeed_update: handle signals consistently in different shellsHiltjo Posthuma
- Handle SIGTERM properly, don't leave stray processes. Kill them on both SIGTERM and SIGINT. - When a "batch" of feeds was interrupted, don't allow to wait again. - Simplify and create sighandler function. - Now on both SIGTERM and SIGINT the cleanup() handler is called to not leave stray files. Tested with ksh, dash, bash, zsh.
2018-10-05sfeed_update: improve SIGINT handlingHiltjo Posthuma
When SIGINT occurs on waiting for jobs it returns 130 (128 + SIGINT). Make sure to check for interrupted and return immediately. Tested with ksh, dash, bash, zsh. Sidenote: ideally we want to cleanup() on SIGTERM too, but this is too inconsistent over various shells.
2018-10-05sfeed_update: small code-style consistency fixesHiltjo Posthuma
2018-09-30sfeed_update: add variable for max amount of feeds to update concurrentlyHiltjo Posthuma
This adds a variable for the maximum amount of feeds to update concurrently. A system/user may have fork resource limits or want to setup some job limit. Thanks leot for the idea and feedback!
2018-09-30sfeed_update: remove -S option (show error), it is not wantedHiltjo Posthuma
2018-09-30sfeed_update: add filter(), order() support per feed + improvementsHiltjo Posthuma
Pass the name parameter to the functions and add these to the pipeline. They can be overridden in the config. - add the ability to change the merge logic per feed. - add the ability to filter lines and fields per feed. - add the ability to order lines differently per feed. - add filter example to README. - code-style: - fetchfeed consistency in parameter order. - change [ x"" = x"" ] to [ "" = "" ]. Simplify some if statements. - wrap long line in fetchfeed(). - use signal names for trap.
2018-09-10sfeed_update: reorder loadconfig and signal handler setupHiltjo Posthuma
... this is useful to change the interrupted behaviour in some use-cases. Thanks leot for the feedback.
2018-09-07sfeed_update: don't always exit 1, exit 130 on SIGINT, exit 0 otherwiseHiltjo Posthuma
Reported by "Dekedro", thanks!
2018-07-18sfeed_update: fail on feed HTTP redirectHiltjo Posthuma
Make curl fail (return a non-zero exit status) on a HTTP redirect. This makes sure sfeed_update shows the feed as "FAILED" instead of succesful with zero data.
2018-02-18sfeed_update: revert more merge logicHiltjo Posthuma
this makes sure the sort order of the initial feed sync works again.
2018-02-18sfeed_update: revert new merge logicHiltjo Posthuma
this requires more work without breaking the order in sfeed_html and other tools (top to bottom: new to oldest), vs sfeed_plain in tail mode: oldest to newest. There will also be improvements to the merge logic to reduce many writes in the future.
2018-02-18sfeed_update: revert previous commit, its not an issueHiltjo Posthuma
2018-02-18sfeed_update: dont allow TAB in filename (messes up the merge logic)Hiltjo Posthuma
2018-02-18sfeed_update: use filename as feedname againHiltjo Posthuma
make the feedname sanitization less strict again.
2018-02-18sfeed_update: run awk in C localeHiltjo Posthuma
2017-12-24sfeed_update: merge improvementsHiltjo Posthuma
does not sort all entries by datetime descending anymore. Now only new entries are appending in a datetime ascending order.
2017-12-16sfeed_update: improvementsHiltjo Posthuma
- clarify code and improve linewrapping. - translate characters in filename (allow /) - add feedname as separate feed name field. - change in which order priority the field is checked.
2016-05-21sfeed_update: unique sort feed on first creationHiltjo Posthuma
when a feed file is created for the first time make sure to sort and filter duplicate items using the same logic as merge().
2016-04-12sfeed_update: fix sorting on fields: id, link, titleHiltjo Posthuma
2016-01-31sfeed_update: fix issue with merging failed feedsHiltjo Posthuma
When fetching a feed failed and its temporary file was stored (filesize=0) it would still be merged with the (possible) old file. This updated the modification time which would be used in the next poll (If-Modified-Since). The solution is to check if the file is non-empty and only then merge, this is still not 100% fool-proof, but much better.
2016-01-31sfeed_update: don't try to sort on feedname field, it was removedHiltjo Posthuma
generated by cgit v1.2.3 (git 2.39.1) at 2024-05-20 03:29:49 +0000