crontab conversion from DA 1.61.5 to DA 1.63.3

ccto

Verified User
Joined
Feb 24, 2005
Messages
192
Location
Hong Kong
We are running conversion from CentOS 8 to AlmaLinux 8.

After OS conversion, we rebuild everything (updated DA, custombuild build , etc)

However, for user crontab file -
The original file /usr/local/directadmin/data/users/xxxx/crontab.conf



During dataskq execution, it throwed a number of -
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
crontab returned non zero value (euid=0): "./data/users/xxxx/crontab.conf.tmp":4: bad minute
errors in crontab file, can't install.

sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
crontab returned non zero value (euid=0): "./data/users/yyyy/crontab.conf.tmp":3: bad minute
errors in crontab file, can't install.


After update, the file /usr/local/directadmin/data/users/xxxx/crontab.conf becomes:
000=sh: which: line 1: syntax error: unexpected end of file
001=sh: error importing function definition for `which'
MAILTO=
PATH=/usr/local/php74/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/xxxx/bin

After update , /var/spool/cron/xxxx
#direct_crons enabled. Safe to edit this file. DirectAdmin will update accordingly.
MAILTO=""
PATH=/usr/local/php74/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/xxxx/bin

(Those DA users are running for years (or may be near 10 years, perhaps from CentOS 4 after a number of migration)

---

What shall I do next? (restore it or else? )
 
Last edited:

Richard G

Verified User
Joined
Jul 6, 2008
Messages
8,527
Location
Maastricht
Do these users by any chance use Drupal? Because I found this:

It's odd that this would happen during a rebuild, but one never knows. Since the errors are exactly the same, I thought I bring it to your attention.
 

ccto

Verified User
Joined
Feb 24, 2005
Messages
192
Location
Hong Kong
Hello Richard G,

Thank you for your link/info.

At my glance, it applied to nearly all users (perhaps except those admin ones) inside my server (including my testing empty account).
 

ccto

Verified User
Joined
Feb 24, 2005
Messages
192
Location
Hong Kong
The problem still exists.

Environment: AlmaLinux 8.5, DirectAdmin v.1.63.3 (latest now) ; custombuild version: 2.0.0 (rev: 2828) , Apache 2.4.51 + mod_fcgid

I do not enable cron for DA users.

Inside /var/spool/cron/todayins
#direct_crons enabled. Safe to edit this file. DirectAdmin will update accordingly.
MAILTO=""
PATH=/usr/local/php74/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/todayins/bin


I manually fixed crontab.conf first (i.e. removed first 2 lines - those sh errors) before re-compilation. Now, it becomes -
MAILTO=
PATH=/usr/local/php74/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/todayins/bin

I issued -
cd /usr/local/directadmin/custombuild
./build update
./build php_expert 8.0 fastcgi

At the end of compilation, it shows -

...
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
sh: which: line 1: syntax error: unexpected end of file
sh: error importing function definition for `which'
crontab returned non zero value (euid=0): "./data/users/todayins/crontab.conf.tmp":4: bad minute
errors in crontab file, can't install.

PHP 8.0.14 Installed.


Basically, it will rewrite httpd config. It applied all DA users.

[[email protected] todayins]# more /usr/local/directadmin/data/users/todayins/crontab.conf
000=sh: which: line 1: syntax error: unexpected end of file
001=sh: error importing function definition for `which'
MAILTO=
PATH=/usr/local/php74/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/todayins/bin
 

ccto

Verified User
Joined
Feb 24, 2005
Messages
192
Location
Hong Kong
The case is
- we do not enable cron for user.
- No cron entries for user.
- dataskq tried to convert the crontab but seems syntax error.
 
Last edited:

DirectAdmin Support

Administrator
Staff member
Joined
Feb 27, 2003
Messages
9,163
Ah, I've seen this once. It was debugged down an issue with the OS itself. As it was only once, we suspected something specific to that box.
If there are more reports, then there's more to it.

For general reference, the ticket in this case is #36411, also using Alma Linux 8.

We narrowed it down to this bug:

and it creeps into the dump of the direct_crons=1 (which everyone uses), where the data/users/fred/crontab.conf is filled based on `crontab -u fred -l`, but only seems to show up when a specific variation of the ENV is used, so duplicating it from the shell was unsuccessful.

I believe it's related to this file:
Code:
/etc/profile.d/which2.sh

Where the ENV DA is using when the crontab.conf rewrite happens contains:
Code:
 0: BASH_FUNC_which%%=() {  ( alias
...
25: which_declare=declare -f

If you type:
Code:
env
in the shell, you'd see the similar thing:
Code:
BASH_FUNC_which%%=() {  ( alias; eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "[email protected]" }

where it seems to be some sort of dynamic function feature, all preceded by BASH_FUNC_... where something about it isn't working correctly.

Now that there's other reports, I've added a temporary workaround to just filter out any crontab output lines that start with "sh: ".
By temporary, I've set a date of June 2022 to remove the code, so AlmaLinux should have definitely sorted it out by then, as I really don't like adding extra complexity to DA code for other bugs, but the DA filtering with at least suppress the issue for now.

Reports on the above URL state that updating the system again should resolve it, but does clarify exactly which steps were taken.
I can speculate that you'd need to:
  1. Update your system with yum, and confirm that the /etc/profile.d/which2.sh has changed.. assumingly to include the } on the same line.
  2. You might need to login to ssh fresh to get the latest ENV vars loaded and restart DA so it gets the new ENV.
  3. Worst case, reboot the box to purge all instances of the bad ENV.
In any case, I've completed the workaround in DA, the alpha binaries should be up after the daily run (we only do it once a day now).
So if you'd like the workaround/fitler, wait until tomorrow, switch to the alpha update channel in your Licenses/Updates page, and grab those binaries. The next time the crontab.conf is re-written (eg: try full ./build rewrite_confs), it should filter out that output.


The other reported workaround solution was to add:
Code:
unset which
to your /etc/bashrc, start a new shell (run "bash") and restart directadmin. The restart might not be required as the crontab call does trigger a fresh shell, but it's plausible that the old ENV vars DA already has might need to be purged. I'm not sure what that affects though, so might be a bit of an agressive hack, in case something needs the "which" function/alias they've setup.

Hopefully this gets everyone affected moving in the right direction. The main solution is to update the box with yum to get a proper fix.. and then flush the env.. likely with a fresh ssh login and DA restart. (reboot if required).

John
 

Richard G

Verified User
Joined
Jul 6, 2008
Messages
8,527
Location
Maastricht
Update your system with yum, and confirm that the /etc/profile.d/which2.sh has changed.. assumingly to include the } on the same line.
On the same line where exactly? When using the env command, I get the same line as you have with the } kind of at the same line.

The which2.sh looks like this on my system (and is from may 5th 2021) and can't be updated as there is no newer version.
Code:
[[email protected]: /etc/profile.d]# less which2.csh
# Initialization script for csh

# alias which 'alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'
So no { or } in there.

The env command gives this output:
Code:
BASH_FUNC_which%%=() {  ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "[email protected]"
}
_=/usr/bin/env
and a lot of stuff before this.

Maybe @ccto can verify he has the same. I'm on Alma 8.5, but didn't see this issue yet, but one never knows.
 

DirectAdmin Support

Administrator
Staff member
Joined
Feb 27, 2003
Messages
9,163
I'm honestly not 100% sure as to "why" the code causes what it does, but I believe it's related to this:
Code:
if [ "$0" = "ksh" ] || [ "$0" = "-ksh" ] ; then
  alias which='(alias; typeset -f) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot'
else
  alias which='(alias; declare -f) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot' fi
and how it's loading it into "$BASH_FUNC_which%%"... somehow missing the closing } character.

As mentioned, I couldn't trigger in from the ssh/shell. It only showed up when DA was making this call "crontab -u fred -l". And something shell related is spitting out the extra errors amongst the crontab file. I'm assuming the aledged yum update fix sorts out the missing } character.. just not 100% sure. I believe you'd know you're affected if your data/users/fred/crontab.conf files have lines like:
Code:
000=sh: which: line 1: syntax error: unexpected end of file
001=sh: error importing function definition for `which'
where the 000/001 are DA's indexes for each cron, and a cron cannot start with "sh: ", it should be starting with "* * * * *" (for example), hence the filtering of the "sh: " lines is a very quick workaround to the problem. Other options are to drop accepting stderr from the output, but as a zero code is being returned DA assumes no stderr should be spit out. The other solution would be to pipe stdout to one buffer, and stderr to it's own, but I've opted for the filtering to get a fix out more quickly, as double-pipe parsing would need a fair amount of testing before I would accept it as working correctly. This bug should be temporary anyway, so the filetering works for now, and I'll schedule the removal of that code sometime in June 2022, on the assumption that everyone's updated their AlmaLinux by then, and assuming it's been fixed and the code isn't needed. If it's still not, we can re-assess then.

I believe they have fixed it already, so not all boxes would be affected. Again, you'll know, as your crontab.conf files would be loaded with the bad cron commands starting with "sh: " instead of a proper * * * * * / @reboot type line.
 

ccto

Verified User
Joined
Feb 24, 2005
Messages
192
Location
Hong Kong
I have 3 servers having this issues (one dev, 2 live). The 2 live servers cloned from this dev image in 2020. (i.e. from CentOS 8.x , then proceed AlmaLinux conversion).

One live box was full yum updated on 19 Dec 2021 and reboot, then custombuild, and throw this error.
One live box was full yum updated on 25 Dec 2021 and reboot, then custombuild, and throw this error.
Both 3 servers encountered the same issue

---

Today, in dev one, I re-run " yum -y update " , "reboot"

After yum update -
[[email protected] profile.d]# md5sum which2.sh
3774e13bd659d7cdbf3f943a95dec506 which2.sh

[[email protected] profile.d]# ll which2.sh
-rw-r--r-- 1 root root 478 May 5 2021 which2.sh


[[email protected] profile.d]# cat which2.sh
# shellcheck shell=sh
# Initialization script for bash, sh, mksh and ksh

which_declare="declare -f"
which_opt="-f"
which_shell="$(cat /proc/$$/comm)"

if [ "$which_shell" = "ksh" ] || [ "$which_shell" = "mksh" ] || [ "$which_shell" = "zsh" ] ; then
which_declare="typeset -f"
which_opt=""
fi

which ()
{
(alias; eval ${which_declare}) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "[email protected]"
}

export which_declare
export ${which_opt} which


env : (last few lines)
LESSOPEN=||/usr/bin/lesspipe.sh %s
BASH_FUNC_which%%=() { ( alias;
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "[email protected]"
}
OLDPWD=/etc/profile.d

Then custombuild , rebuild php , same crontab errors for each DA users.

(i.e. after full yum update, it still has problem)

---

Finally, per suggestion, I applied temporary workaround ( i.e. appended "unset which" to /root/.bashrc , not /etc/bashrc ) , and reboot

It works in dev environment. (No such crontab errors during custombuild build/rewrite.)

(Not sure any side effects)

Anyway, thank you for your help first.
 
Last edited:

ccto

Verified User
Joined
Feb 24, 2005
Messages
192
Location
Hong Kong
Perhaps, the minimal-side-effort workaround:

cd /usr/local/directadmin/custombuild
./build update

unset which ; ./build php_expert 8.0 fastcgi

...

. /etc/profile.d/which2.sh

It worked.
 

Richard G

Verified User
Joined
Jul 6, 2008
Messages
8,527
Location
Maastricht
and how it's loading it into "$BASH_FUNC_which%%"... somehow missing the closing } character.
But the closing end character is at the next line in the output.

I'll have look if I got any users with crontabs on the server and check if the problem exists here too. Can take some time since it's old years day. :)
 
Top