summaryrefslogtreecommitdiff
path: root/sfeed_update
AgeCommit message (Collapse)Author
2023-06-09sfeed_update/sfeedrc: add url as parameter to the filter() and order() functionHiltjo Posthuma
This might make it easier to set filters or ordering by pattern matching on a group of feeds by the feed URL. For example youtube or reddit feeds. Another way which was already possible is prefixing names with for example: "reddit somename" or "yt somename".
2023-04-05sfeed_update: fail early if creating a temporary directory or status file failsHiltjo Posthuma
If creating a temporary directory for the feed files failed then $sfeedtmpdir would be empty and it would try to: mkdir -p "/feed" touch "/feed/ok" After failing it would also still try to process all the feeds. Now just fail early. mktemp or touch themselve will print the actual error to stderr.
2023-02-17sfeed_update, sfeed_opml_export, README: reference the example sfeedrc man pageHiltjo Posthuma
... and some small rewording.
2022-06-21fix some typisNRK
found via codespell $ codespell --ignore-regex Nd
2022-03-28sfeed_update: change return to exit in mainHiltjo Posthuma
Pedantic change: Make main more consistent since other functions in it exit too and main is not supposed to return or used like that.
2022-03-25change echo to printf and for sfeed_opml_export use a control-character ↵Hiltjo Posthuma
separator echo is unportable in this way and names containing characters like an option (-n) or backslash or escape codes (\e, \n, etc) could be messy. For awk set LC_ALL=C for simple collation. This makes sfeed_opml_export slower in some shells that don't have printf builtin though. For example with about 150 feeds in a config file it is a bit slower on OpenBSD ksh. time ./sfeed_opml_export | wc -l 152 0m00.29s real 0m00.05s user 0m00.20s system time sfeed_opml_export | wc -l 152 0m00.02s real 0m00.00s user 0m00.03s system
2022-03-23shellscripts: use [ for test consistentlyHiltjo Posthuma
2022-03-22sfeed_update: return status in _feed() functionHiltjo Posthuma
This can be useful for scripts, for example the sfeed_update_xargs example script in the README. This way the process can signal an error and xargs will exit with the code 123: "One or more invocations of utility returned a nonzero exit status."
2022-03-21sfeed_update: set exit status non-zero if any of the feeds failedHiltjo Posthuma
In practise this may change the meaning of the examples: sfeed_update && pkill -SIGHUP sfeed_curses An alternative: sfeed_update; pkill -SIGHUP sfeed_curses
2022-03-21sfeed_update: log FAILs to stderrTommy Nguyen
2021-05-27sfeed_update: fix message when the configuration file does not existHiltjo Posthuma
When sfeed_update was called without using a parameter and it used the default and this path did not exist it would incorrectly print: Configuration file "" does not exist or is not readable. See sfeedrc.example for an example. Make the error message a bit shorter too. This was a partial regression of commit df74ba274c4ea5d9b7388c33500ba601ed0c991d
2021-03-03sfeed_update: return instead of exit in main() on successHiltjo Posthuma
This is useful so the script can be included, call main and then have additional post-main functionality.
2021-03-01sfeed_update: fix baseurl substitutionHiltjo Posthuma
Follow-up from a rushed commit: commit 58555779d123be68c0acf9ea898931d656ec6d63 Author: Hiltjo Posthuma <hiltjo@codemadness.org> Date: Sun Feb 28 13:33:21 2021 +0100 sfeed_update: simplify, use feedurl directly This also make it possible to use non-authoritive URLs as a baseurl, like "magnet:" URLs.
2021-03-01sfeed_update: simplify, use feedurl directlyHiltjo Posthuma
This also make it possible to use non-authoritive URLs as a baseurl, like "magnet:" URLs.
2021-02-05sfeed_update: $SFEED_UPDATE_INCLUDE: be a bit more precise/pedanticHiltjo Posthuma
2021-01-27sfeed_update: $SFEED_UPDATE_INCLUDE: be a bit more precise/pedanticHiltjo Posthuma
2021-01-27sfeed_update: allow to reuse the code more easily as an included scriptHiltjo Posthuma
This adds a main() function. When the environment variable $SFEED_UPDATE_INCLUDE is set then it will not execute the main handler. The other functions are included and can be reused. This is also useful for unit-testing.
2021-01-27sfeed_update: separate code of parallel exection and feed() into a _feed() ↵Hiltjo Posthuma
handler This is useful to be able to reuse the code (together with using sfeed_update as an included script, coming in the next commit).
2021-01-27sfeed_update: shuffle code getting the path of the feedurl to make the ↵Hiltjo Posthuma
basesiteurl Move it closer before it is used.
2021-01-27sfeed_update: change parse failure error messageHiltjo Posthuma
"(FAIL CONVERT)" -> "(FAIL PARSE)". Convert may be too similar to text encoding conversion.
2021-01-27sfeed_update: add an overridable parse() function, using sfeed(1) by defaultHiltjo Posthuma
This can be useful to make more cleanly make connector scripts. This does not necesarily even have to be in the sfeed(5) format.
2021-01-24sfeed_update: print the filename again as passed as a parameterHiltjo Posthuma
... and do not show stderr of readlink.
2021-01-16sfeed_update: typo in commentHiltjo Posthuma
2021-01-16sfeed_update: improve consistency of feed creation and mergingHiltjo Posthuma
- Improve feed creation with empty results and new feed files. Always make sure the file is created even when it is new and there are also no items (after filtering). - Consistency: always use the same feed file for merging. Do not use "/dev/null" when it is a new file. This works using sort, but is ugly when the merge() function is overridden and does something else. It should be the feed file always.
2021-01-16sfeed_update: make convertencoding() consistent with other overridable functionsHiltjo Posthuma
This adds the name as the first parameter for the convertencoding() function, like filter, merge, order, etc. This can be useful to make an exception rule for text decoding in a more clean way.
2021-01-01sfeed_update: if baseurl is empty then use the path from the feed by defaultHiltjo Posthuma
Feeds should contain absolute urls, but if it does not have it then this makes it more convenient to configure such feeds.
2020-02-23sfeed_update: don't preserve permissions of tmp files by moving, so copyHiltjo Posthuma
noticed on DragonFlyBSD where it prints a warning when moving the file from /tmp. To reproduce it: touch /tmp/file mv /tmp/file ~/ On other systems this would not print a warning, but it would preserve the group permissions etc.
2020-02-04sfeed_update: tiny style fix in commentHiltjo Posthuma
2019-06-11sfeed_update: clarify a comment about being non-optimalHiltjo Posthuma
2019-05-15README and sfeed_update: use names (order vs sort) and in the execution orderHiltjo Posthuma
2019-05-12sfeed_update: cleanup proper encoding file (if any)Hiltjo Posthuma
2019-05-02sfeed_update: disable If-Modified-Since by defaultHiltjo Posthuma
2019-04-14sfeed_update: fix typo in commentHiltjo Posthuma
2019-04-14sfeed_update: rename fetchfeed to fetchHiltjo Posthuma
... and simplify example in README.
2019-04-14sfeed_update improvementsHiltjo Posthuma
- Better checking and verbose logging (on failure) of each stage: fetchfeed, filter, merge, order, convertencoding. This makes sure on out-of-memory, disk-space or other resource limits the output is not corrupted. - This also has the added advantage it runs less processes (piped) at the same time. - Clear previous unneeded file to preserve space in /tmp (/tmp is often mounted as mfs/tmpfs). - Add logging function (able to override), use more logical logging format (pun intended). - Code-style: order overridable functions in execution order.
2019-03-01sfeed_update: remove wrong/incomplete commentHiltjo Posthuma
2018-10-24sfeed_update: fix wrong comment "temporary file" -> "temporary directory"Hiltjo Posthuma
2018-10-24Revert "sfeed_update: replace non-POSIX mktemp with $$"Hiltjo Posthuma
This reverts commit 8699fa2bb4c75670952fee503a58ca4a652627eb. There is a regression in directory permissions among other things.
2018-10-11sfeed_update: replace non-POSIX mktemp with $$Hiltjo Posthuma
+ fix wrong comment "temporary file" -> "temporary directory".
2018-10-06sfeed_update: handle signals consistently in different shellsHiltjo Posthuma
- Handle SIGTERM properly, don't leave stray processes. Kill them on both SIGTERM and SIGINT. - When a "batch" of feeds was interrupted, don't allow to wait again. - Simplify and create sighandler function. - Now on both SIGTERM and SIGINT the cleanup() handler is called to not leave stray files. Tested with ksh, dash, bash, zsh.
2018-10-05sfeed_update: improve SIGINT handlingHiltjo Posthuma
When SIGINT occurs on waiting for jobs it returns 130 (128 + SIGINT). Make sure to check for interrupted and return immediately. Tested with ksh, dash, bash, zsh. Sidenote: ideally we want to cleanup() on SIGTERM too, but this is too inconsistent over various shells.
2018-10-05sfeed_update: small code-style consistency fixesHiltjo Posthuma
2018-09-30sfeed_update: add variable for max amount of feeds to update concurrentlyHiltjo Posthuma
This adds a variable for the maximum amount of feeds to update concurrently. A system/user may have fork resource limits or want to setup some job limit. Thanks leot for the idea and feedback!
2018-09-30sfeed_update: remove -S option (show error), it is not wantedHiltjo Posthuma
2018-09-30sfeed_update: add filter(), order() support per feed + improvementsHiltjo Posthuma
Pass the name parameter to the functions and add these to the pipeline. They can be overridden in the config. - add the ability to change the merge logic per feed. - add the ability to filter lines and fields per feed. - add the ability to order lines differently per feed. - add filter example to README. - code-style: - fetchfeed consistency in parameter order. - change [ x"" = x"" ] to [ "" = "" ]. Simplify some if statements. - wrap long line in fetchfeed(). - use signal names for trap.
2018-09-10sfeed_update: reorder loadconfig and signal handler setupHiltjo Posthuma
... this is useful to change the interrupted behaviour in some use-cases. Thanks leot for the feedback.
2018-09-07sfeed_update: don't always exit 1, exit 130 on SIGINT, exit 0 otherwiseHiltjo Posthuma
Reported by "Dekedro", thanks!
2018-07-18sfeed_update: fail on feed HTTP redirectHiltjo Posthuma
Make curl fail (return a non-zero exit status) on a HTTP redirect. This makes sure sfeed_update shows the feed as "FAILED" instead of succesful with zero data.
2018-02-18sfeed_update: revert more merge logicHiltjo Posthuma
this makes sure the sort order of the initial feed sync works again.
2018-02-18sfeed_update: revert new merge logicHiltjo Posthuma
this requires more work without breaking the order in sfeed_html and other tools (top to bottom: new to oldest), vs sfeed_plain in tail mode: oldest to newest. There will also be improvements to the merge logic to reduce many writes in the future.
generated by cgit v1.2.3 (git 2.39.1) at 2024-10-16 16:23:26 +0000