summaryrefslogtreecommitdiff
path: root/sfeed_update
AgeCommit message (Collapse)Author
2018-10-11sfeed_update: replace non-POSIX mktemp with $$Hiltjo Posthuma
+ fix wrong comment "temporary file" -> "temporary directory".
2018-10-06sfeed_update: handle signals consistently in different shellsHiltjo Posthuma
- Handle SIGTERM properly, don't leave stray processes. Kill them on both SIGTERM and SIGINT. - When a "batch" of feeds was interrupted, don't allow to wait again. - Simplify and create sighandler function. - Now on both SIGTERM and SIGINT the cleanup() handler is called to not leave stray files. Tested with ksh, dash, bash, zsh.
2018-10-05sfeed_update: improve SIGINT handlingHiltjo Posthuma
When SIGINT occurs on waiting for jobs it returns 130 (128 + SIGINT). Make sure to check for interrupted and return immediately. Tested with ksh, dash, bash, zsh. Sidenote: ideally we want to cleanup() on SIGTERM too, but this is too inconsistent over various shells.
2018-10-05sfeed_update: small code-style consistency fixesHiltjo Posthuma
2018-09-30sfeed_update: add variable for max amount of feeds to update concurrentlyHiltjo Posthuma
This adds a variable for the maximum amount of feeds to update concurrently. A system/user may have fork resource limits or want to setup some job limit. Thanks leot for the idea and feedback!
2018-09-30sfeed_update: remove -S option (show error), it is not wantedHiltjo Posthuma
2018-09-30sfeed_update: add filter(), order() support per feed + improvementsHiltjo Posthuma
Pass the name parameter to the functions and add these to the pipeline. They can be overridden in the config. - add the ability to change the merge logic per feed. - add the ability to filter lines and fields per feed. - add the ability to order lines differently per feed. - add filter example to README. - code-style: - fetchfeed consistency in parameter order. - change [ x"" = x"" ] to [ "" = "" ]. Simplify some if statements. - wrap long line in fetchfeed(). - use signal names for trap.
2018-09-10sfeed_update: reorder loadconfig and signal handler setupHiltjo Posthuma
... this is useful to change the interrupted behaviour in some use-cases. Thanks leot for the feedback.
2018-09-07sfeed_update: don't always exit 1, exit 130 on SIGINT, exit 0 otherwiseHiltjo Posthuma
Reported by "Dekedro", thanks!
2018-07-18sfeed_update: fail on feed HTTP redirectHiltjo Posthuma
Make curl fail (return a non-zero exit status) on a HTTP redirect. This makes sure sfeed_update shows the feed as "FAILED" instead of succesful with zero data.
2018-02-18sfeed_update: revert more merge logicHiltjo Posthuma
this makes sure the sort order of the initial feed sync works again.
2018-02-18sfeed_update: revert new merge logicHiltjo Posthuma
this requires more work without breaking the order in sfeed_html and other tools (top to bottom: new to oldest), vs sfeed_plain in tail mode: oldest to newest. There will also be improvements to the merge logic to reduce many writes in the future.
2018-02-18sfeed_update: revert previous commit, its not an issueHiltjo Posthuma
2018-02-18sfeed_update: dont allow TAB in filename (messes up the merge logic)Hiltjo Posthuma
2018-02-18sfeed_update: use filename as feedname againHiltjo Posthuma
make the feedname sanitization less strict again.
2018-02-18sfeed_update: run awk in C localeHiltjo Posthuma
2017-12-24sfeed_update: merge improvementsHiltjo Posthuma
does not sort all entries by datetime descending anymore. Now only new entries are appending in a datetime ascending order.
2017-12-16sfeed_update: improvementsHiltjo Posthuma
- clarify code and improve linewrapping. - translate characters in filename (allow /) - add feedname as separate feed name field. - change in which order priority the field is checked.
2016-05-21sfeed_update: unique sort feed on first creationHiltjo Posthuma
when a feed file is created for the first time make sure to sort and filter duplicate items using the same logic as merge().
2016-04-12sfeed_update: fix sorting on fields: id, link, titleHiltjo Posthuma
2016-01-31sfeed_update: fix issue with merging failed feedsHiltjo Posthuma
When fetching a feed failed and its temporary file was stored (filesize=0) it would still be merged with the (possible) old file. This updated the modification time which would be used in the next poll (If-Modified-Since). The solution is to check if the file is non-empty and only then merge, this is still not 100% fool-proof, but much better.
2016-01-31sfeed_update: don't try to sort on feedname field, it was removedHiltjo Posthuma
2015-09-23sfeed_update: dont send user-agent by default (curl+version) (privacy/security)Hiltjo Posthuma
2015-08-22sfeed_update: simplify path (only $sfeedpath now)Hiltjo Posthuma
2015-08-14minor code-style improvementsHiltjo Posthuma
2015-08-14sfeed_update: don't redirect, use only the feed url specified.Hiltjo Posthuma
It could mess up urls in items (redirect http to https). It is also safer.
2015-08-07sfeed_update: just use the feed name as the temporary filenameHiltjo Posthuma
the base temporary directory is random. The directory is cleaned afterwards or on SIGTERM etc so remove this unneeded line.
2015-08-05sfeed_update: fix time modified on feed first runHiltjo Posthuma
A feeds file was created with touch when it didnt exist. This confuses curl -z since it uses that file for If-Modified-Since. Now it just copies the temporary file to the new file without merging if it doesn't exist. The warning for curl for a non-existant file is already suppressed with curl -s.
2015-08-05sfeed_update, opml_export: slightly safer comparisonHiltjo Posthuma
2015-07-31improve documentationHiltjo Posthuma
2015-06-21change feeds file format, its now more consistentHiltjo Posthuma
update sfeed_update, there is now a feeds file per feed.
2015-05-25sfeed_update: dont sort on feedurlHiltjo Posthuma
feed urls sometimes change and is not important for the order.
2015-01-28sfeed_update: pass file for curl time condition (last modified)Hiltjo Posthuma
this fixes an issue with differences in stat(1) versions (OpenBSD, GNU, sbase).
2015-01-05sfeed_update: wrap long lineHiltjo Posthuma
2015-01-04sfeed_update: styleHiltjo Posthuma
2014-11-11sfeed_update: remove redundant -t flag for mktempHiltjo Posthuma
this makes it compatible with the current sbase mktemp
2014-04-08sfeed_update: small time format, timeout from 30 to 15 secondsHiltjo Posthuma
Signed-off-by: Hiltjo Posthuma <hiltjo@codemadness.org>
2014-03-31new versionHiltjo Posthuma
lots of things changed, but cleanup todo. changelog and consistent stream of small updates will come in the future. Signed-off-by: Hiltjo Posthuma <hiltjo@codemadness.org>
2013-05-20add basesiteurl, simplify sfeed_update a bitHiltjo Posthuma
Signed-off-by: Hiltjo Posthuma <hiltjo@codemadness.org>
2012-08-03Add base url of a site parameter for sfeedrcHiltjo Posthuma
This allows to have a feed on a different domain but specify the base url of links if links in the feed are relative. Signed-off-by: Hiltjo Posthuma <hiltjo@codemadness.org>
2012-08-03New initial repoHiltjo Posthuma
Signed-off-by: Hiltjo Posthuma <hiltjo@codemadness.org>
generated by cgit v1.2.3 (git 2.39.1) at 2024-10-16 15:33:57 +0000