You can subscribe to this list here.
| 2002 |
Jan
|
Feb
|
Mar
|
Apr
(23) |
May
(45) |
Jun
(22) |
Jul
(11) |
Aug
(14) |
Sep
(38) |
Oct
(62) |
Nov
(34) |
Dec
(25) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2003 |
Jan
(88) |
Feb
(126) |
Mar
(182) |
Apr
(168) |
May
(244) |
Jun
(249) |
Jul
(206) |
Aug
(309) |
Sep
(250) |
Oct
(487) |
Nov
(620) |
Dec
(470) |
| 2004 |
Jan
(623) |
Feb
(671) |
Mar
(522) |
Apr
(841) |
May
(671) |
Jun
(638) |
Jul
(622) |
Aug
(544) |
Sep
(580) |
Oct
(621) |
Nov
(668) |
Dec
(575) |
| 2005 |
Jan
(440) |
Feb
(547) |
Mar
(626) |
Apr
(594) |
May
(597) |
Jun
(796) |
Jul
(760) |
Aug
(908) |
Sep
(948) |
Oct
(923) |
Nov
(1114) |
Dec
(686) |
| 2006 |
Jan
(897) |
Feb
(943) |
Mar
(1049) |
Apr
(882) |
May
(774) |
Jun
(736) |
Jul
(816) |
Aug
(798) |
Sep
(1000) |
Oct
(742) |
Nov
(881) |
Dec
(705) |
| 2007 |
Jan
(1086) |
Feb
(1085) |
Mar
(1132) |
Apr
(1011) |
May
(778) |
Jun
(847) |
Jul
(857) |
Aug
(865) |
Sep
(830) |
Oct
(939) |
Nov
(856) |
Dec
(543) |
| 2008 |
Jan
(933) |
Feb
(832) |
Mar
(772) |
Apr
(587) |
May
(723) |
Jun
(872) |
Jul
(962) |
Aug
(915) |
Sep
(766) |
Oct
(658) |
Nov
(780) |
Dec
(554) |
| 2009 |
Jan
(604) |
Feb
(766) |
Mar
(719) |
Apr
(745) |
May
(547) |
Jun
(554) |
Jul
(474) |
Aug
(338) |
Sep
(424) |
Oct
(670) |
Nov
(421) |
Dec
(510) |
| 2010 |
Jan
(732) |
Feb
(702) |
Mar
(693) |
Apr
(666) |
May
(556) |
Jun
(515) |
Jul
(553) |
Aug
(549) |
Sep
(344) |
Oct
(431) |
Nov
(437) |
Dec
(329) |
| 2011 |
Jan
(822) |
Feb
(540) |
Mar
(435) |
Apr
(437) |
May
(624) |
Jun
(458) |
Jul
(416) |
Aug
(395) |
Sep
(333) |
Oct
(280) |
Nov
(246) |
Dec
(324) |
| 2012 |
Jan
(340) |
Feb
(273) |
Mar
(429) |
Apr
(321) |
May
(311) |
Jun
(329) |
Jul
(201) |
Aug
(307) |
Sep
(263) |
Oct
(308) |
Nov
(315) |
Dec
(294) |
| 2013 |
Jan
(481) |
Feb
(337) |
Mar
(310) |
Apr
(269) |
May
(274) |
Jun
(231) |
Jul
(182) |
Aug
(214) |
Sep
(276) |
Oct
(178) |
Nov
(222) |
Dec
(150) |
| 2014 |
Jan
(135) |
Feb
(144) |
Mar
(218) |
Apr
(152) |
May
(312) |
Jun
(187) |
Jul
(197) |
Aug
(218) |
Sep
(241) |
Oct
(282) |
Nov
(292) |
Dec
(229) |
| 2015 |
Jan
(200) |
Feb
(133) |
Mar
(154) |
Apr
(162) |
May
(268) |
Jun
(274) |
Jul
(166) |
Aug
(311) |
Sep
(182) |
Oct
(236) |
Nov
(160) |
Dec
(216) |
| 2016 |
Jan
(187) |
Feb
(248) |
Mar
(259) |
Apr
(112) |
May
(203) |
Jun
(104) |
Jul
(156) |
Aug
(131) |
Sep
(135) |
Oct
(161) |
Nov
(179) |
Dec
(110) |
| 2017 |
Jan
(148) |
Feb
(96) |
Mar
(236) |
Apr
(99) |
May
(118) |
Jun
(156) |
Jul
(157) |
Aug
(204) |
Sep
(151) |
Oct
(152) |
Nov
(125) |
Dec
(58) |
| 2018 |
Jan
(127) |
Feb
(151) |
Mar
(119) |
Apr
(131) |
May
(170) |
Jun
(125) |
Jul
(103) |
Aug
(119) |
Sep
(143) |
Oct
(116) |
Nov
(141) |
Dec
(90) |
| 2019 |
Jan
(179) |
Feb
(126) |
Mar
(97) |
Apr
(135) |
May
(135) |
Jun
(110) |
Jul
(121) |
Aug
(61) |
Sep
(96) |
Oct
(48) |
Nov
(58) |
Dec
(105) |
| 2020 |
Jan
(116) |
Feb
(97) |
Mar
(114) |
Apr
(96) |
May
(154) |
Jun
(116) |
Jul
(76) |
Aug
(20) |
Sep
(68) |
Oct
(105) |
Nov
(33) |
Dec
(118) |
| 2021 |
Jan
(34) |
Feb
(81) |
Mar
(94) |
Apr
(74) |
May
(133) |
Jun
(86) |
Jul
(65) |
Aug
(44) |
Sep
(68) |
Oct
(56) |
Nov
(113) |
Dec
(195) |
| 2022 |
Jan
(135) |
Feb
(65) |
Mar
(108) |
Apr
(48) |
May
(102) |
Jun
(153) |
Jul
(89) |
Aug
(90) |
Sep
(135) |
Oct
(77) |
Nov
(85) |
Dec
(61) |
| 2023 |
Jan
(102) |
Feb
(62) |
Mar
(81) |
Apr
(103) |
May
(71) |
Jun
(45) |
Jul
(57) |
Aug
(60) |
Sep
(94) |
Oct
(104) |
Nov
(96) |
Dec
(68) |
| 2024 |
Jan
(107) |
Feb
(92) |
Mar
(91) |
Apr
(155) |
May
(78) |
Jun
(121) |
Jul
(64) |
Aug
(136) |
Sep
(108) |
Oct
(105) |
Nov
(124) |
Dec
(88) |
| 2025 |
Jan
(115) |
Feb
(95) |
Mar
(84) |
Apr
(23) |
May
(59) |
Jun
(89) |
Jul
(71) |
Aug
(59) |
Sep
(60) |
Oct
(24) |
Nov
(56) |
Dec
(56) |
|
From: Marcin H. <gan...@gm...> - 2025-12-25 12:27:03
|
On Wed, 24 Dec 2025 at 14:20, Martin Simmons <ma...@li...> wrote: > > >>>>> On Wed, 24 Dec 2025 07:02:57 +0100, Marcin Haba said: > > > > Bacularis uses Bconsole in the same way as Bacula administrator uses > > it. I don't see a direct relation between Bacularis and the Bacula > > Director segfault. > > Does Bacularis send console commands that are rarely typed by a human > administrator? That is a possible difference. Hello Martin, For the way of using Bconsole I mean how Bconsole is used, that Bacularis executes supported Bconsole commands the same as they are used by administrator and as a separate app does not cause Director segfault directly. There are executed different Bconsole commands both these used by users and these more suitable for scripts and programs. Best regards, Marcin Haba (gani) |
|
From: Martin S. <ma...@li...> - 2025-12-24 19:29:26
|
The "affected_rows=0" is the more detailed error information here (but that is generated by Bacula itself). This is not a problem that Postgresql will give more information about because an SQL command that updates nothing is perfectly OK in general. It just means that the WHERE clause didn't match anything. __Martin >>>>> On Tue, 23 Dec 2025 13:50:27 -0600, kjohnson said: > > Hi, > > Again, thanks for the interest in my problem. I was certainly not expecting a patch for this old version of Bacula. This system will get a Bacula version update when the update to Debian occurs, according to the local policy governing this system. I actually did not assume that there was a Bacula bug here, though I could argue that there is a bug of not reporting more detailed error information from Postgresql -- I am sure that Postgresql provides more detailed information than 'Update failed'. > > It's too late to run the SQL command you suggested. The dbcheck run mentioned in my original post would have deleted it if it was present. > > I have not seen this error recur. If it does, I will look into the tracing you mention. While I have no direct experience with Bacula trace files, it does not seem that it would be impossible to use tools to find a database error, and then perhaps see the error returned by Postgres. I could be wrong about that. I might also be mistaken in my thinking that knowing why the update failed would be helpful. > > Best regards, > > Ken > > > -----Original Message----- > From: bac...@li... [mailto:bac...@li...] > Sent: Thursday, December 11, 2025 2:13 PM > To: bac...@li... > Subject: Re: [Bacula-users] Troubleshoot bdb.h fatal error? > > Hi Ken, > > Am 11.12.2025 um 20:26 schrieb kjo...@ec...: > > Rob, Arno, > > > > Thank you for taking an interest in my problem. > > You're welcome! > > Looks like the simple, obvious things do not help us here. So... > > > Answers to questions, as best as I can provide: > > > >> from Rob: > >> You mentioned that the last two admin jobs failed. Was that a typo? If not, what errors did the last job (unmount, eject) give? > > > > The errors for jobid 27943 look very much like the errors for 27941. > > > > 08-Dec 14:21 linux2-dir JobId 27943: Fatal error: bdb.h:140 Update failed: affected_rows=0 for UPDATE Job SET JobStatus='R',Level=' ',StartTime='2025-12-08 14:21:57',ClientId=1,JobTDate=1765225317,PoolId=0,FileSetId=0 WHERE JobId=27943 > > 08-Dec 14:21 linux2-dir JobId 27943: Fatal error: bdb.h:140 Update failed: affected_rows=0 for UPDATE Job SET JobStatus='f',Level=' ',StartTime='2025-12-08 14:21:57',ClientId=1,JobTDate=1765225317,PoolId=0,FileSetId=0 WHERE JobId=27943 > > We#ll need to find out what failed here. There is a simple possibility > for the catalog update to fail, that is when the row its supposed to > update does not exist. > > In bconsole, do > > sql > select * from job where jobid=27943; > > and see if it finds that row. > > If it doesn't, I'm wondering why the fact that such a job could not be > created was not reported -- it should have been. > > > 08-Dec 14:21 linux2-dir JobId 27943: Warning: Error updating job record. bdb.h:140 Update failed: affected_rows=0 for UPDATE Job SET JobStatus='f',EndTime='2025-12-08 14:21:57',ClientId=1,JobBytes=0,ReadBytes=0,JobFiles=0,JobErrors=1,VolSessionId=0,VolSessionTime=0,PoolId=0,FileSetId=0,JobTDate=1765225317,RealEndTime='2025-12-08 14:21:57',PriorJobId=0,HasBase=0,PurgedFiles=0 WHERE JobId=27943 > > 08-Dec 14:21 linux2-dir JobId 27943: Warning: Error getting Job record for Job report: ERR=sql_get.c:303 No Job found for JobId 27943 > > We can probably guess the result of above exercise, but let's not guess :-) > > > 08-Dec 14:21 linux2-dir JobId 27943: Error: Bacula 9.6.7 (10Dec20): 08-Dec-2025 14:21:57 > > So we would have to investigate if the DIR for some reason "forgot" to > create a job record when the job was started (I have never experienced > such a thing, but that doesn't prove anything), if it didn't log it for > some reason, if you just missed the error message (that would be > convenient in this case :-) or if something deleted it in between > successful job creation and the first update. > > Debugging, as a user, something that did *not* happen is a bit of a > challenge, but we can probably achieve something if you can reproduce > the problem. > > However, we'll probably not be able to convince Eric and team to fix > issues in version 9 anymore. > > Thus -- would you be able to upgrade to a recent version, preferrbla the > most recent one? > > I would recommend using the packages you can subscribe to at > https://www.bacula.org/bacula-binary-package-download/ but, if that's > not a choice you would consider, building from source is also an option. > Proper packaging is above my pay grade, though :-) > > The alternative to enable tracing, debug, reproduce and eventually > carefully read a few million lines of traces files will probably get us > somewhere, but will not actually solve anything... > > Cheers, > > Arno > > -- > Arno Lehmann > > IT-Service Lehmann > Sandstr. 6, 49080 Osnabrück > > > > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users > |
|
From: Martin S. <ma...@li...> - 2025-12-24 13:20:13
|
>>>>> On Wed, 24 Dec 2025 07:02:57 +0100, Marcin Haba said: > > Bacularis uses Bconsole in the same way as Bacula administrator uses > it. I don't see a direct relation between Bacularis and the Bacula > Director segfault. Does Bacularis send console commands that are rarely typed by a human administrator? That is a possible difference. __Martin |
|
From: Martin S. <ma...@li...> - 2025-12-24 13:16:32
|
>>>>> On Wed, 24 Dec 2025 06:07:33 +0100, Marcin Haba said: > > On Tue, 23 Dec 2025 at 21:46, Martin Simmons <ma...@li...> wrote: > > Note that value=0xaaaaaaaaaaaaaaaa, which is a pattern glibc puts in freed > > memory, so looks like a use-after-free bug. This is the value of user->host() > > at the end of handle_UA_client_request, but its not clear how that would be > > freed. > > Hello Everybody, > > I can be wrong but for me it looks like a possible problem in > bvsnprintf() function. > > In this output: > > #4 0x00007ff005024d59 in fmtstr (buffer=buffer@entry=0x7fefcc00f890 > "Disconnection from 226.144.140:9101", currlen=currlen@entry=19, > maxlen=maxlen@entry=512, value=0xaaaaaaaaaaaaaaaa <error: Cannot > access memory at address 0xaaaaaaaaaaaaaaaa>, flags=0, min=0, max=512) > at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/bsnprintf.c:462 > #5 0x00007ff005025995 in bvsnprintf > (buffer=buffer@entry=0x7fefcc00f890 "Disconnection from > 226.144.140:9101", maxlen=512, format=<optimized out>, > format@entry=0x55b590a39128 "Disconnection from %s:%d", > args=args@entry=0x7fefe9ffab10) at > /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/bsnprintf.c:362 > > > the IP address looks to be incomplete (3 octets): "Disconnection from > 226.144.140:9101". > > The currlen=currlen@entry=19 indicates to position 19 which is the > start position of the IP address: "Disconnection from " Yes, so I think that means it hasn't copied the IP address into the buffer yet. It calls fmtstr to copy it, which is where we see the argument value=0xaaaaaaaaaaaaaaaa and the crash. Unfortunately, gdb doesn't show the varargs passed to bvsnprintf, so we can only assume that it was passed 0xaaaaaaaaaaaaaaaa for that argument. A likely explanation for the incomplete IP address is that we are seeing the old contents of the buffer from offset 19 onwards. I suspect the buffer was used previously for the "Connection from %s:%d" message near the start of handle_UA_client_request. If you overlay these two messages and assume that the first octet of the IP address has two digits, then offset 19 would refer to bbb.ccc.ddd:ppp before it is set by fmtstr: Connection from aa.bbb.ccc.ddd:ppp Disconnection from aa.bbb.ccc.ddd:ppp __Martin |
|
From: Marcin H. <gan...@gm...> - 2025-12-24 06:03:20
|
On Wed, 24 Dec 2025 at 02:42, Martin Juhl Prendergast <m...@rt...> wrote: > [root@degobah mj]# rpm -qa |grep bareos > > But, I am using Bacularis.. so this might be some kind of issue there?? > Hello Martin, Bacularis uses Bconsole in the same way as Bacula administrator uses it. I don't see a direct relation between Bacularis and the Bacula Director segfault. Best regards, Marcin Haba (gani) |
|
From: Marcin H. <gan...@gm...> - 2025-12-24 05:07:56
|
On Tue, 23 Dec 2025 at 21:46, Martin Simmons <ma...@li...> wrote: > Note that value=0xaaaaaaaaaaaaaaaa, which is a pattern glibc puts in freed > memory, so looks like a use-after-free bug. This is the value of user->host() > at the end of handle_UA_client_request, but its not clear how that would be > freed. Hello Everybody, I can be wrong but for me it looks like a possible problem in bvsnprintf() function. In this output: #4 0x00007ff005024d59 in fmtstr (buffer=buffer@entry=0x7fefcc00f890 "Disconnection from 226.144.140:9101", currlen=currlen@entry=19, maxlen=maxlen@entry=512, value=0xaaaaaaaaaaaaaaaa <error: Cannot access memory at address 0xaaaaaaaaaaaaaaaa>, flags=0, min=0, max=512) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/bsnprintf.c:462 #5 0x00007ff005025995 in bvsnprintf (buffer=buffer@entry=0x7fefcc00f890 "Disconnection from 226.144.140:9101", maxlen=512, format=<optimized out>, format@entry=0x55b590a39128 "Disconnection from %s:%d", args=args@entry=0x7fefe9ffab10) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/bsnprintf.c:362 the IP address looks to be incomplete (3 octets): "Disconnection from 226.144.140:9101". The currlen=currlen@entry=19 indicates to position 19 which is the start position of the IP address: "Disconnection from " Maybe something happened here... I am curious what is the problem :-) and if this is the right path (I am not a C programmer). Best regards, Marcin Haba (gani) |
|
From: Martin J. P. <m...@rt...> - 2025-12-24 01:41:02
|
I can't think what it should be.. I was comming from Bareos before going to Bacula, but there doesn't seem to be anything left: [root@degobah mj]# rpm -qa |grep bacula bacularis-httpd-5.0.0-1.el9.noarch bacularis-5.0.0-1.el9.noarch bacularis-selinux-5.0.0-1.el9.noarch bacula-libs-15.0.3-3.el9.x86_64 bacula-common-15.0.3-3.el9.x86_64 bacula-libs-sql-15.0.3-3.el9.x86_64 bacula-director-15.0.3-3.el9.x86_64 bacula-storage-15.0.3-3.el9.x86_64 bacula-client-15.0.3-3.el9.x86_64 bacula-console-15.0.3-3.el9.x86_64 bacula-debuginfo-15.0.3-3.el9.x86_64 bacula-director-debuginfo-15.0.3-3.el9.x86_64 bacula-libs-debuginfo-15.0.3-3.el9.x86_64 bacula-libs-sql-debuginfo-15.0.3-3.el9.x86_64 [root@degobah mj]# rpm -qa |grep bareos But, I am using Bacularis.. so this might be some kind of issue there?? Another thing.. the backup did succeed two days ago, and have been running stable ever since (because it's only running incremental backups now).. It also was weird that sometimes it would run for hours and sometimes only for 5 minutes.. Any ideas? Regards Martin On Tirsdag, December 23, 2025 22:11 CET, Rob Gerber <ro...@cr...> wrote: > I wonder if any old components from previous bacula installations (if any) > are still resident on the system. I also wonder the same thing about any > remnants of the previous bareos installation > > In particular, I wonder if there are any old tray monitors or GUI > applications still around. > > Robert Gerber > 402-237-8692 > ro...@cr... > > > > > > |
|
From: Martin J. P. <m...@rt...> - 2025-12-24 01:36:47
|
Hi Martin The console connections are probably because i'm running Bacularis.. Which uses the console for accessing data.. Maybe there is some kind of limit on console connections that breaks?? I got the source from here: https://kojipkgs.fedoraproject.org//packages/bacula/15.0.3/3.fc44/src/bacula-15.0.3-3.fc44.src.rpm But I had (apparently) the same issue on the standard bacula 11 from EPEL9.. Regards Martin On Tirsdag, December 23, 2025 21:44 CET, Martin Simmons <ma...@li...> wrote: > This is the backtrace of the crash: > > Thread 21 (Thread 0x7fefe9ffb640 (LWP 3927517) "bacula-dir"): > #0 0x00007ff0048d9fff in wait4 () from /lib64/libc.so.6 > #1 0x00007ff00505015c in signal_handler (sig=11) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/signal.c:229 > #2 <signal handler called> > #3 0x00007ff0049666fd in __strlen_avx2_rtm () from /lib64/libc.so.6 > #4 0x00007ff005024d59 in fmtstr (buffer=buffer@entry=0x7fefcc00f890 "Disconnection from 226.144.140:9101", currlen=currlen@entry=19, maxlen=maxlen@entry=512, value=0xaaaaaaaaaaaaaaaa <error: Cannot access memory at address 0xaaaaaaaaaaaaaaaa>, flags=0, min=0, max=512) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/bsnprintf.c:462 > #5 0x00007ff005025995 in bvsnprintf (buffer=buffer@entry=0x7fefcc00f890 "Disconnection from 226.144.140:9101", maxlen=512, format=<optimized out>, format@entry=0x55b590a39128 "Disconnection from %s:%d", args=args@entry=0x7fefe9ffab10) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/bsnprintf.c:362 > #6 0x000055b5909e1df2 in UAContext::send_events (this=0x7fefbc00d068, code=0x55b590a39141 "DC0016", type=0x55b590a39116 "connection", fmt=0x55b590a39128 "Disconnection from %s:%d") at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/dird/ua_output.c:1475 > #7 0x000055b590a003b9 in handle_UA_client_request (arg=0x7feff4040bf8) at ../lib/bsockcore.h:168 > #8 0x00007ff00505ac9b in workq_server (arg=arg@entry=0x55b590a5aac0 <ua_workq>) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/workq.c:372 > #9 0x00007ff00506a902 in lmgr_thread_launcher (x=0x7feff400d728) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/lockmgr.c:1189 > #10 0x00007ff00488b2ea in start_thread () from /lib64/libc.so.6 > #11 0x00007ff0049103c0 in clone3 () from /lib64/libc.so.6 > > Note that value=0xaaaaaaaaaaaaaaaa, which is a pattern glibc puts in freed > memory, so looks like a use-after-free bug. This is the value of user->host() > at the end of handle_UA_client_request, but its not clear how that would be > freed. > > One interesting thing is that this a "console" connection, not anything > directly related to a job. There appears to be 5 console connections, all > created around the same time: > > -Console-.2025-12-20_00.51.12_03 > -Console-.2025-12-20_00.51.16_37 > -Console-.2025-12-20_00.51.16_38 > -Console-.2025-12-20_00.51.16_39 > -Console-.2025-12-20_00.51.16_40 > > The crash was apparently detected by bacula-dir at 00:51:13 (but it might have > taken a few seconds for gdb to start). > > Are those console connections something you expect from your setup? > > Do you have a link to the source code you compiled (15.0.3-3.el9)? > > __Martin > > > >>>>> On Sat, 20 Dec 2025 01:03:09 +0100, Martin Juhl Prendergast said: > > > > Ok.. I finally get a real traceback: > > > > https://pastebin.com/sz7uWiYM > > > > Hope someone wiser than me can make some sense of it.. > > > > /Martin > > > > On Fredag, December 19, 2025 21:04 CET, Martin Simmons <ma...@li...> wrote: > > > > > The error: > > > > > > 'fail_time' has unknown type; cast it to its declared type > > > > > > means that gdb can't find any symbolic debugging information for bacula-dir > > > and/or libbac. If you installed bacula from rpm packages, then that > > > information is probably stripped out. > > > > > > Did the bacula compilation generate any debuginfo packages? If so, try > > > installing those as well. > > > > > > __Martin > > > > > > > > > >>>>> On Fri, 19 Dec 2025 20:20:42 +0100, Martin Juhl Prendergast said: > > > > > > > > Hi > > > > > > > > I'm not sure that I got gdb to work???: > > > > > > > > Check the log files for more information. > > > > > > > > [New LWP 3745058] > > > > [New LWP 3745057] > > > > [New LWP 3745056] > > > > [New LWP 3745055] > > > > [New LWP 3745054] > > > > [New LWP 3745053] > > > > [New LWP 3745052] > > > > [New LWP 3745051] > > > > [New LWP 3745034] > > > > [New LWP 3745033] > > > > [New LWP 3745032] > > > > [New LWP 3745031] > > > > [New LWP 3745030] > > > > [New LWP 3745029] > > > > [New LWP 3745025] > > > > [New LWP 3745024] > > > > [New LWP 3744994] > > > > [New LWP 3744993] > > > > [New LWP 3744988] > > > > [New LWP 3744981] > > > > [New LWP 3069995] > > > > [New LWP 3069994] > > > > [New LWP 3069985] > > > > [New LWP 3069968] > > > > [New LWP 3069967] > > > > [New LWP 3069965] > > > > [New LWP 3069963] > > > > [New LWP 3069962] > > > > [New LWP 3060166] > > > > [New LWP 3060113] > > > > [New LWP 2985337] > > > > [New LWP 2985336] > > > > [New LWP 2985335] > > > > [New LWP 2985332] > > > > [Thread debugging using libthread_db enabled] > > > > Using host libthread_db library "/lib64/libthread_db.so.1". > > > > 0x00007f968dc8837a in __futex_abstimed_wait_common () from /lib64/libc.so.6 > > > > /usr/libexec/bacula/btraceback.gdb:1: Error in sourced command file: > > > > 'fail_time' has unknown type; cast it to its declared type > > > > [Inferior 1 (process 2985331) detached] > > > > Attempt to dump locks > > > > threadid=0x7f9606ffd640 max=1 current=-1 > > > > threadid=0x7f9627fff640 max=1 current=-1 > > > > threadid=0x7f9684ff9640 max=1 current=-1 > > > > threadid=0x7f96857fa640 max=1 current=-1 > > > > threadid=0x7f9666ffd640 max=1 current=-1 > > > > threadid=0x7f9667fff640 max=1 current=-1 > > > > threadid=0x7f9664ff9640 max=1 current=-1 > > > > threadid=0x7f9605ffb640 max=1 current=-1 > > > > threadid=0x7f96277fe640 max=1 current=-1 > > > > threadid=0x7f96477fe640 max=1 current=-1 > > > > threadid=0x7f96657fa640 max=1 current=-1 > > > > threadid=0x7f9646ffd640 max=1 current=-1 > > > > threadid=0x7f96077fe640 max=1 current=-1 > > > > threadid=0x7f96467fc640 max=1 current=-1 > > > > threadid=0x7f96677fe640 max=1 current=-1 > > > > threadid=0x7f96067fc640 max=1 current=-1 > > > > threadid=0x7f9607fff640 max=2 current=-1 > > > > threadid=0x7f9645ffb640 max=1 current=-1 > > > > threadid=0x7f9647fff640 max=1 current=-1 > > > > threadid=0x7f9665ffb640 max=1 current=-1 > > > > threadid=0x7f9624ff9640 max=2 current=-1 > > > > threadid=0x7f96257fa640 max=2 current=-1 > > > > threadid=0x7f9625ffb640 max=2 current=-1 > > > > threadid=0x7f9685ffb640 max=2 current=-1 > > > > threadid=0x7f9644ff9640 max=2 current=-1 > > > > threadid=0x7f96457fa640 max=2 current=-1 > > > > threadid=0x7f96867fc640 max=2 current=-1 > > > > threadid=0x7f96667fc640 max=2 current=-1 > > > > threadid=0x7f96267fc640 max=2 current=-1 > > > > threadid=0x7f9626ffd640 max=2 current=-1 > > > > threadid=0x7f9686ffd640 max=1 current=-1 > > > > threadid=0x7f96877fe640 max=2 current=-1 > > > > threadid=0x7f9687fff640 max=0 current=-1 > > > > threadid=0x7f968cfaa640 max=0 current=-1 > > > > threadid=0x7f968de12f40 max=1 current=-1 > > > > Attempt to dump current JCRs. njcrs=7 > > > > threadid=0x7f968de12f40 JobId=0 JobStatus=R jcr=0x55eae96c6da8 name=*JobMonitor*.2025-12-18_22.57.48_01 > > > > use_count=1 killable=0 > > > > JobType=I JobLevel= > > > > sched_time=18-Dec-2025 22:57 start_time=18-Dec-2025 22:57 > > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > wstore=0x55eae95f64e8 rstore=(nil) wjcr=(nil) client=0x55eae95f3788 reschedule_count=0 SD_msg_chan_started=0 > > > > threadid=0x7f9626ffd640 JobId=61 JobStatus=R jcr=0x7f963c015e88 name=SullustBackup.2025-12-19_00.53.46_20 > > > > use_count=2 killable=1 > > > > JobType=B JobLevel=F > > > > sched_time=19-Dec-2025 00:53 start_time=19-Dec-2025 00:53 > > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > > db=0x7f963c024e88 db_batch=(nil) batch_started=0 > > > > wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=1 > > > > BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true > > > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384 > > > > RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0 > > > > threadid=0x7f9686ffd640 JobId=67 JobStatus=c jcr=0x7f967c01c908 name=SullustBackup.2025-12-19_01.00.00_17 > > > > use_count=1 killable=0 > > > > JobType=B JobLevel=F > > > > sched_time=19-Dec-2025 01:00 start_time=19-Dec-2025 01:00 > > > > end_time=01-Jan-1970 01:00 wait_time=19-Dec-2025 01:00 > > > > db=0x7f963c024e88 db_batch=(nil) batch_started=0 > > > > wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=0 > > > > BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true > > > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384 > > > > RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0 > > > > threadid=0x7f9607fff640 JobId=0 JobStatus=R jcr=0x7f962800b328 name=-Console-.2025-12-19_20.11.56_53 > > > > use_count=1 killable=0 > > > > JobType=U JobLevel= > > > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > > > threadid=0x7f9665ffb640 JobId=0 JobStatus=R jcr=0x7f962c00b6f8 name=-Console-.2025-12-19_20.11.57_20 > > > > use_count=1 killable=0 > > > > JobType=U JobLevel= > > > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > > > threadid=0x7f9606ffd640 JobId=0 JobStatus=R jcr=0x7f963c00f5f8 name=-Console-.2025-12-19_20.11.58_37 > > > > use_count=1 killable=0 > > > > JobType=U JobLevel= > > > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > > > threadid=0x7f9627fff640 JobId=0 JobStatus=R jcr=0x7f963800db68 name=-Console-.2025-12-19_20.11.58_38 > > > > use_count=1 killable=0 > > > > JobType=U JobLevel= > > > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > > > List plugins. Hook count=0 > > > > > > > > On Torsdag, December 18, 2025 15:38 CET, Martin Simmons <ma...@li...> wrote: > > > > > > > > > You will need to install gdb as well, so it can get backtraces. > > > > > > > > > > __Martin > > > > > > > > > > > > > > > >>>>> On Thu, 18 Dec 2025 14:41:46 +0100, Martin Juhl Prendergast said: > > > > > > > > > > > > Hi Arno > > > > > > > > > > > > Traceback is inserted below.. > > > > > > > > > > > > Original I installed the packages from the EPEL9 repository, but when I had the problem there, I rebuilt the 15.0.3 package from Fedora 44, on RHEL9... only to see the same issue.. > > > > > > > > > > > > I have started the storage daemon in debug mode, and is waiting to see debug for that, the next time it crashes.. > > > > > > > > > > > > Regards > > > > > > > > > > > > If you need any more info, please > > > > > > > > > > > > Check the log files for more information. > > > > > > > > > > > > Please install a debugger (gdb) to receive a traceback. > > > > > > Attempt to dump locks > > > > > > threadid=0x7fc5477fe640 max=1 current=-1 > > > > > > threadid=0x7fc564ff9640 max=1 current=-1 > > > > > > threadid=0x7fc5467fc640 max=1 current=-1 > > > > > > threadid=0x7fc547fff640 max=2 current=-1 > > > > > > threadid=0x7fc5457fa640 max=1 current=-1 > > > > > > threadid=0x7fc545ffb640 max=2 current=-1 > > > > > > threadid=0x7fc59d80e640 max=1 current=-1 > > > > > > threadid=0x7fc59e00f640 max=2 current=-1 > > > > > > threadid=0x7fc59e810640 max=0 current=-1 > > > > > > threadid=0x7fc59f1ff640 max=0 current=-1 > > > > > > threadid=0x7fc5a000bf40 max=1 current=-1 > > > > > > Attempt to dump current JCRs. njcrs=3 > > > > > > threadid=0x7fc5a000bf40 JobId=0 JobStatus=R jcr=0x558b1d983008 name=*JobMonitor*.2025-12-18_11.30.53_01 > > > > > > use_count=1 killable=0 > > > > > > JobType=I JobLevel= > > > > > > sched_time=18-Dec-2025 11:30 start_time=18-Dec-2025 11:30 > > > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > > > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > > > > > > threadid=0x7fc545ffb640 JobId=54 JobStatus=R jcr=0x7fc5700187b8 name=SullustBackup.2025-12-18_11.35.50_45 > > > > > > use_count=2 killable=1 > > > > > > JobType=B JobLevel=F > > > > > > sched_time=18-Dec-2025 11:35 start_time=18-Dec-2025 11:35 > > > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > > > db=0x7fc570027348 db_batch=(nil) batch_started=0 > > > > > > wstore=0x558b1d8afab8 rstore=(nil) wjcr=(nil) client=0x558b1d8ad5e8 reschedule_count=0 SD_msg_chan_started=1 > > > > > > BDB=0x7fc570027348 db_name=bacula db_user=bacula connected=true > > > > > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=16 > > > > > > RWLOCK=0x7fc570027360 w_active=0 w_wait=0 > > > > > > threadid=0x7fc547fff640 JobId=0 JobStatus=R jcr=0x7fc53400b098 name=-Console-.2025-12-18_11.36.53_00 > > > > > > use_count=1 killable=0 > > > > > > JobType=U JobLevel= > > > > > > sched_time=18-Dec-2025 11:36 start_time=18-Dec-2025 11:36 > > > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > > > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > > > > > > List plugins. Hook count=0 > > > > > > > > > > > > > > > > > > On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <bac...@li...> wrote: > > > > > > > > > > > > > Hi Martin, > > > > > > > > > > > > > > Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast: > > > > > > > > Using Debug on bacula-dir, I get this: > > > > > > > > > > > > > > > > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25) > > > > > > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan > > > > > > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > > > > > > > > > > > Definitely deserves a thorough investigation. It's unlilkely to be > > > > > > > caused by configuration. > > > > > > > > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958] > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/ > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked... > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback > > > > > > > > > > > > > > That file will now be relevant. > > > > > > > > > > > > > > /var/spool/bacula/bacula.2464062.traceback should contain information > > > > > > > the developers can work with. > > > > > > > > > > > > > > Also interesting will be to know where you installed the packages from, > > > > > > > or how you built the software. > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Arno > > > > > > > > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'. > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1. > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director. > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director. > > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected > > > > > > > > > > > > > > > > > > > > > > > > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > > > > > > > > > > > > >> Also.. on the server running bacula/bacularis I get: > > > > > > > >> > > > > > > > >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000] > > > > > > > >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping. > > > > > > > >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000] > > > > > > > >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0) > > > > > > > >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00 > > > > > > > >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000] > > > > > > > >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0) > > > > > > > >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > > > > > >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000] > > > > > > > >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000] > > > > > > > >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping. > > > > > > > >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000] > > > > > > > >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000] > > > > > > > >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0) > > > > > > > >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > > > > > >> > > > > > > > >> > > > > > > > >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > > > > >> > > > > > > > >>> Oh, I can see that.. > > > > > > > >>> > > > > > > > >>> The storage daemon says: > > > > > > > >>> > > > > > > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > > > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. > > > > > > > >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > > > > >>> > > > > > > > >>> Config: > > > > > > > >>> > > > > > > > >>> Director { > > > > > > > >>> Name = "bacula-dir" > > > > > > > >>> Password = "@@SD_PASSWORD@@" > > > > > > > >>> } > > > > > > > >>> Director { > > > > > > > >>> Name = "bacula-mon" > > > > > > > >>> Password = "@@MON_SD_PASSWORD@@" > > > > > > > >>> Monitor = yes > > > > > > > >>> } > > > > > > > >>> Storage { > > > > > > > >>> Name = "bacula-sd" > > > > > > > >>> WorkingDirectory = "/var/spool/bacula" > > > > > > > >>> PidDirectory = "/var/run" > > > > > > > >>> PluginDirectory = "/usr/lib64/bacula" > > > > > > > >>> MaximumConcurrentJobs = 20 > > > > > > > >>> } > > > > > > > >>> Device { > > > > > > > >>> Name = "AlwaysIncrement" > > > > > > > >>> Description = "" > > > > > > > >>> MediaType = "AlwaysIncrement" > > > > > > > >>> DeviceType = "File" > > > > > > > >>> ArchiveDevice = "/home/bacula/autoincrement" > > > > > > > >>> RemovableMedia = no > > > > > > > >>> RandomAccess = yes > > > > > > > >>> AutomaticMount = yes > > > > > > > >>> LabelMedia = yes > > > > > > > >>> Autochanger = no > > > > > > > >>> ReadOnly = no > > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > > >>> DriveIndex = 0 > > > > > > > >>> } > > > > > > > >>> Device { > > > > > > > >>> Name = "FileChgr1-Dev1" > > > > > > > >>> MediaType = "File1" > > > > > > > >>> ArchiveDevice = "/tmp" > > > > > > > >>> RemovableMedia = no > > > > > > > >>> RandomAccess = yes > > > > > > > >>> AutomaticMount = yes > > > > > > > >>> LabelMedia = yes > > > > > > > >>> AlwaysOpen = no > > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > > >>> } > > > > > > > >>> Device { > > > > > > > >>> Name = "FileChgr1-Dev2" > > > > > > > >>> MediaType = "File1" > > > > > > > >>> ArchiveDevice = "/tmp" > > > > > > > >>> RemovableMedia = no > > > > > > > >>> RandomAccess = yes > > > > > > > >>> AutomaticMount = yes > > > > > > > >>> LabelMedia = yes > > > > > > > >>> AlwaysOpen = no > > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > > >>> } > > > > > > > >>> Device { > > > > > > > >>> Name = "FileChgr2-Dev1" > > > > > > > >>> MediaType = "File2" > > > > > > > >>> ArchiveDevice = "/tmp" > > > > > > > >>> RemovableMedia = no > > > > > > > >>> RandomAccess = yes > > > > > > > >>> AutomaticMount = yes > > > > > > > >>> LabelMedia = yes > > > > > > > >>> AlwaysOpen = no > > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > > >>> } > > > > > > > >>> Device { > > > > > > > >>> Name = "FileChgr2-Dev2" > > > > > > > >>> MediaType = "File2" > > > > > > > >>> ArchiveDevice = "/tmp" > > > > > > > >>> RemovableMedia = no > > > > > > > >>> RandomAccess = yes > > > > > > > >>> AutomaticMount = yes > > > > > > > >>> LabelMedia = yes > > > > > > > >>> AlwaysOpen = no > > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > > >>> } > > > > > > > >>> Messages { > > > > > > > >>> Name = "Standard" > > > > > > > >>> Director = bacula-dir = All > > > > > > > >>> } > > > > > > > >>> Autochanger { > > > > > > > >>> Name = "FileChgr1" > > > > > > > >>> Device = "FileChgr1-Dev1" > > > > > > > >>> Device = "FileChgr1-Dev2" > > > > > > > >>> ChangerDevice = "/dev/null" > > > > > > > >>> ChangerCommand = "" > > > > > > > >>> } > > > > > > > >>> Autochanger { > > > > > > > >>> Name = "FileChgr2" > > > > > > > >>> Device = "FileChgr2-Dev1" > > > > > > > >>> Device = "FileChgr2-Dev2" > > > > > > > >>> ChangerDevice = "/dev/null" > > > > > > > >>> ChangerCommand = "" > > > > > > > >>> } > > > > > > > >>> Device { > > > > > > > >>> DeviceType = "File" > > > > > > > >>> RemovableMedia = no > > > > > > > >>> AutomaticMount = yes > > > > > > > >>> LabelMedia = yes > > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > > >>> RandomAccess = yes > > > > > > > >>> Name = "Consolidate" > > > > > > > >>> Description = "" > > > > > > > >>> DriveIndex = 0 > > > > > > > >>> ArchiveDevice = "/home/bacula/consolidate" > > > > > > > >>> MediaType = "Consolidate" > > > > > > > >>> ReadOnly = no > > > > > > > >>> Autochanger = no > > > > > > > >>> } > > > > > > > >>> > > > > > > > >>> Please say if I need to provide more configuration > > > > > > > >>> > > > > > > > >>> /Martin > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> Martin, > > > > > > > >>> > > > > > > > >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says". > > > > > > > >>> > > > > > > > >>> Regards, > > > > > > > >>> Robert Gerber > > > > > > > >>> 402-237-8692 > > > > > > > >>> ro...@cr... > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: > > > > > > > >>> > > > > > > > >>> Hi guys > > > > > > > >>> > > > > > > > >>> Hope someone can help me.. > > > > > > > >>> > > > > > > > >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. > > > > > > > >>> > > > > > > > >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. > > > > > > > >>> > > > > > > > >>> On the hosts I get: > > > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. > > > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer > > > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> The Storage daemon says > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> _______________________________________________ > > > > > > > >>> Bacula-users mailing list > > > > > > > >>> Bac...@li... > > > > > > > >>> https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > Bacula-users mailing list > > > > > > > > Bac...@li... > > > > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > -- > > > > > > > Arno Lehmann > > > > > > > > > > > > > > IT-Service Lehmann > > > > > > > Sandstr. 6, 49080 Osnabrück > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Bacula-users mailing list > > > > > > > Bac...@li... > > > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Bacula-users mailing list > > > > > > Bac...@li... > > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > > > > > |
|
From: Rob G. <ro...@cr...> - 2025-12-23 21:35:59
|
I wonder if any old components from previous bacula installations (if any) are still resident on the system. I also wonder the same thing about any remnants of the previous bareos installation In particular, I wonder if there are any old tray monitors or GUI applications still around. Robert Gerber 402-237-8692 ro...@cr... > > |
|
From: Martin S. <ma...@li...> - 2025-12-23 20:45:13
|
This is the backtrace of the crash: Thread 21 (Thread 0x7fefe9ffb640 (LWP 3927517) "bacula-dir"): #0 0x00007ff0048d9fff in wait4 () from /lib64/libc.so.6 #1 0x00007ff00505015c in signal_handler (sig=11) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/signal.c:229 #2 <signal handler called> #3 0x00007ff0049666fd in __strlen_avx2_rtm () from /lib64/libc.so.6 #4 0x00007ff005024d59 in fmtstr (buffer=buffer@entry=0x7fefcc00f890 "Disconnection from 226.144.140:9101", currlen=currlen@entry=19, maxlen=maxlen@entry=512, value=0xaaaaaaaaaaaaaaaa <error: Cannot access memory at address 0xaaaaaaaaaaaaaaaa>, flags=0, min=0, max=512) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/bsnprintf.c:462 #5 0x00007ff005025995 in bvsnprintf (buffer=buffer@entry=0x7fefcc00f890 "Disconnection from 226.144.140:9101", maxlen=512, format=<optimized out>, format@entry=0x55b590a39128 "Disconnection from %s:%d", args=args@entry=0x7fefe9ffab10) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/bsnprintf.c:362 #6 0x000055b5909e1df2 in UAContext::send_events (this=0x7fefbc00d068, code=0x55b590a39141 "DC0016", type=0x55b590a39116 "connection", fmt=0x55b590a39128 "Disconnection from %s:%d") at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/dird/ua_output.c:1475 #7 0x000055b590a003b9 in handle_UA_client_request (arg=0x7feff4040bf8) at ../lib/bsockcore.h:168 #8 0x00007ff00505ac9b in workq_server (arg=arg@entry=0x55b590a5aac0 <ua_workq>) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/workq.c:372 #9 0x00007ff00506a902 in lmgr_thread_launcher (x=0x7feff400d728) at /usr/src/debug/bacula-15.0.3-3.el9.x86_64/src/lib/lockmgr.c:1189 #10 0x00007ff00488b2ea in start_thread () from /lib64/libc.so.6 #11 0x00007ff0049103c0 in clone3 () from /lib64/libc.so.6 Note that value=0xaaaaaaaaaaaaaaaa, which is a pattern glibc puts in freed memory, so looks like a use-after-free bug. This is the value of user->host() at the end of handle_UA_client_request, but its not clear how that would be freed. One interesting thing is that this a "console" connection, not anything directly related to a job. There appears to be 5 console connections, all created around the same time: -Console-.2025-12-20_00.51.12_03 -Console-.2025-12-20_00.51.16_37 -Console-.2025-12-20_00.51.16_38 -Console-.2025-12-20_00.51.16_39 -Console-.2025-12-20_00.51.16_40 The crash was apparently detected by bacula-dir at 00:51:13 (but it might have taken a few seconds for gdb to start). Are those console connections something you expect from your setup? Do you have a link to the source code you compiled (15.0.3-3.el9)? __Martin >>>>> On Sat, 20 Dec 2025 01:03:09 +0100, Martin Juhl Prendergast said: > > Ok.. I finally get a real traceback: > > https://pastebin.com/sz7uWiYM > > Hope someone wiser than me can make some sense of it.. > > /Martin > > On Fredag, December 19, 2025 21:04 CET, Martin Simmons <ma...@li...> wrote: > > > The error: > > > > 'fail_time' has unknown type; cast it to its declared type > > > > means that gdb can't find any symbolic debugging information for bacula-dir > > and/or libbac. If you installed bacula from rpm packages, then that > > information is probably stripped out. > > > > Did the bacula compilation generate any debuginfo packages? If so, try > > installing those as well. > > > > __Martin > > > > > > >>>>> On Fri, 19 Dec 2025 20:20:42 +0100, Martin Juhl Prendergast said: > > > > > > Hi > > > > > > I'm not sure that I got gdb to work???: > > > > > > Check the log files for more information. > > > > > > [New LWP 3745058] > > > [New LWP 3745057] > > > [New LWP 3745056] > > > [New LWP 3745055] > > > [New LWP 3745054] > > > [New LWP 3745053] > > > [New LWP 3745052] > > > [New LWP 3745051] > > > [New LWP 3745034] > > > [New LWP 3745033] > > > [New LWP 3745032] > > > [New LWP 3745031] > > > [New LWP 3745030] > > > [New LWP 3745029] > > > [New LWP 3745025] > > > [New LWP 3745024] > > > [New LWP 3744994] > > > [New LWP 3744993] > > > [New LWP 3744988] > > > [New LWP 3744981] > > > [New LWP 3069995] > > > [New LWP 3069994] > > > [New LWP 3069985] > > > [New LWP 3069968] > > > [New LWP 3069967] > > > [New LWP 3069965] > > > [New LWP 3069963] > > > [New LWP 3069962] > > > [New LWP 3060166] > > > [New LWP 3060113] > > > [New LWP 2985337] > > > [New LWP 2985336] > > > [New LWP 2985335] > > > [New LWP 2985332] > > > [Thread debugging using libthread_db enabled] > > > Using host libthread_db library "/lib64/libthread_db.so.1". > > > 0x00007f968dc8837a in __futex_abstimed_wait_common () from /lib64/libc.so.6 > > > /usr/libexec/bacula/btraceback.gdb:1: Error in sourced command file: > > > 'fail_time' has unknown type; cast it to its declared type > > > [Inferior 1 (process 2985331) detached] > > > Attempt to dump locks > > > threadid=0x7f9606ffd640 max=1 current=-1 > > > threadid=0x7f9627fff640 max=1 current=-1 > > > threadid=0x7f9684ff9640 max=1 current=-1 > > > threadid=0x7f96857fa640 max=1 current=-1 > > > threadid=0x7f9666ffd640 max=1 current=-1 > > > threadid=0x7f9667fff640 max=1 current=-1 > > > threadid=0x7f9664ff9640 max=1 current=-1 > > > threadid=0x7f9605ffb640 max=1 current=-1 > > > threadid=0x7f96277fe640 max=1 current=-1 > > > threadid=0x7f96477fe640 max=1 current=-1 > > > threadid=0x7f96657fa640 max=1 current=-1 > > > threadid=0x7f9646ffd640 max=1 current=-1 > > > threadid=0x7f96077fe640 max=1 current=-1 > > > threadid=0x7f96467fc640 max=1 current=-1 > > > threadid=0x7f96677fe640 max=1 current=-1 > > > threadid=0x7f96067fc640 max=1 current=-1 > > > threadid=0x7f9607fff640 max=2 current=-1 > > > threadid=0x7f9645ffb640 max=1 current=-1 > > > threadid=0x7f9647fff640 max=1 current=-1 > > > threadid=0x7f9665ffb640 max=1 current=-1 > > > threadid=0x7f9624ff9640 max=2 current=-1 > > > threadid=0x7f96257fa640 max=2 current=-1 > > > threadid=0x7f9625ffb640 max=2 current=-1 > > > threadid=0x7f9685ffb640 max=2 current=-1 > > > threadid=0x7f9644ff9640 max=2 current=-1 > > > threadid=0x7f96457fa640 max=2 current=-1 > > > threadid=0x7f96867fc640 max=2 current=-1 > > > threadid=0x7f96667fc640 max=2 current=-1 > > > threadid=0x7f96267fc640 max=2 current=-1 > > > threadid=0x7f9626ffd640 max=2 current=-1 > > > threadid=0x7f9686ffd640 max=1 current=-1 > > > threadid=0x7f96877fe640 max=2 current=-1 > > > threadid=0x7f9687fff640 max=0 current=-1 > > > threadid=0x7f968cfaa640 max=0 current=-1 > > > threadid=0x7f968de12f40 max=1 current=-1 > > > Attempt to dump current JCRs. njcrs=7 > > > threadid=0x7f968de12f40 JobId=0 JobStatus=R jcr=0x55eae96c6da8 name=*JobMonitor*.2025-12-18_22.57.48_01 > > > use_count=1 killable=0 > > > JobType=I JobLevel= > > > sched_time=18-Dec-2025 22:57 start_time=18-Dec-2025 22:57 > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > db=(nil) db_batch=(nil) batch_started=0 > > > wstore=0x55eae95f64e8 rstore=(nil) wjcr=(nil) client=0x55eae95f3788 reschedule_count=0 SD_msg_chan_started=0 > > > threadid=0x7f9626ffd640 JobId=61 JobStatus=R jcr=0x7f963c015e88 name=SullustBackup.2025-12-19_00.53.46_20 > > > use_count=2 killable=1 > > > JobType=B JobLevel=F > > > sched_time=19-Dec-2025 00:53 start_time=19-Dec-2025 00:53 > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > db=0x7f963c024e88 db_batch=(nil) batch_started=0 > > > wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=1 > > > BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true > > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384 > > > RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0 > > > threadid=0x7f9686ffd640 JobId=67 JobStatus=c jcr=0x7f967c01c908 name=SullustBackup.2025-12-19_01.00.00_17 > > > use_count=1 killable=0 > > > JobType=B JobLevel=F > > > sched_time=19-Dec-2025 01:00 start_time=19-Dec-2025 01:00 > > > end_time=01-Jan-1970 01:00 wait_time=19-Dec-2025 01:00 > > > db=0x7f963c024e88 db_batch=(nil) batch_started=0 > > > wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=0 > > > BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true > > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384 > > > RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0 > > > threadid=0x7f9607fff640 JobId=0 JobStatus=R jcr=0x7f962800b328 name=-Console-.2025-12-19_20.11.56_53 > > > use_count=1 killable=0 > > > JobType=U JobLevel= > > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > db=(nil) db_batch=(nil) batch_started=0 > > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > > threadid=0x7f9665ffb640 JobId=0 JobStatus=R jcr=0x7f962c00b6f8 name=-Console-.2025-12-19_20.11.57_20 > > > use_count=1 killable=0 > > > JobType=U JobLevel= > > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > db=(nil) db_batch=(nil) batch_started=0 > > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > > threadid=0x7f9606ffd640 JobId=0 JobStatus=R jcr=0x7f963c00f5f8 name=-Console-.2025-12-19_20.11.58_37 > > > use_count=1 killable=0 > > > JobType=U JobLevel= > > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > db=(nil) db_batch=(nil) batch_started=0 > > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > > threadid=0x7f9627fff640 JobId=0 JobStatus=R jcr=0x7f963800db68 name=-Console-.2025-12-19_20.11.58_38 > > > use_count=1 killable=0 > > > JobType=U JobLevel= > > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > > db=(nil) db_batch=(nil) batch_started=0 > > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > > List plugins. Hook count=0 > > > > > > On Torsdag, December 18, 2025 15:38 CET, Martin Simmons <ma...@li...> wrote: > > > > > > > You will need to install gdb as well, so it can get backtraces. > > > > > > > > __Martin > > > > > > > > > > > > >>>>> On Thu, 18 Dec 2025 14:41:46 +0100, Martin Juhl Prendergast said: > > > > > > > > > > Hi Arno > > > > > > > > > > Traceback is inserted below.. > > > > > > > > > > Original I installed the packages from the EPEL9 repository, but when I had the problem there, I rebuilt the 15.0.3 package from Fedora 44, on RHEL9... only to see the same issue.. > > > > > > > > > > I have started the storage daemon in debug mode, and is waiting to see debug for that, the next time it crashes.. > > > > > > > > > > Regards > > > > > > > > > > If you need any more info, please > > > > > > > > > > Check the log files for more information. > > > > > > > > > > Please install a debugger (gdb) to receive a traceback. > > > > > Attempt to dump locks > > > > > threadid=0x7fc5477fe640 max=1 current=-1 > > > > > threadid=0x7fc564ff9640 max=1 current=-1 > > > > > threadid=0x7fc5467fc640 max=1 current=-1 > > > > > threadid=0x7fc547fff640 max=2 current=-1 > > > > > threadid=0x7fc5457fa640 max=1 current=-1 > > > > > threadid=0x7fc545ffb640 max=2 current=-1 > > > > > threadid=0x7fc59d80e640 max=1 current=-1 > > > > > threadid=0x7fc59e00f640 max=2 current=-1 > > > > > threadid=0x7fc59e810640 max=0 current=-1 > > > > > threadid=0x7fc59f1ff640 max=0 current=-1 > > > > > threadid=0x7fc5a000bf40 max=1 current=-1 > > > > > Attempt to dump current JCRs. njcrs=3 > > > > > threadid=0x7fc5a000bf40 JobId=0 JobStatus=R jcr=0x558b1d983008 name=*JobMonitor*.2025-12-18_11.30.53_01 > > > > > use_count=1 killable=0 > > > > > JobType=I JobLevel= > > > > > sched_time=18-Dec-2025 11:30 start_time=18-Dec-2025 11:30 > > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > > > > > threadid=0x7fc545ffb640 JobId=54 JobStatus=R jcr=0x7fc5700187b8 name=SullustBackup.2025-12-18_11.35.50_45 > > > > > use_count=2 killable=1 > > > > > JobType=B JobLevel=F > > > > > sched_time=18-Dec-2025 11:35 start_time=18-Dec-2025 11:35 > > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > > db=0x7fc570027348 db_batch=(nil) batch_started=0 > > > > > wstore=0x558b1d8afab8 rstore=(nil) wjcr=(nil) client=0x558b1d8ad5e8 reschedule_count=0 SD_msg_chan_started=1 > > > > > BDB=0x7fc570027348 db_name=bacula db_user=bacula connected=true > > > > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=16 > > > > > RWLOCK=0x7fc570027360 w_active=0 w_wait=0 > > > > > threadid=0x7fc547fff640 JobId=0 JobStatus=R jcr=0x7fc53400b098 name=-Console-.2025-12-18_11.36.53_00 > > > > > use_count=1 killable=0 > > > > > JobType=U JobLevel= > > > > > sched_time=18-Dec-2025 11:36 start_time=18-Dec-2025 11:36 > > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > > > > > List plugins. Hook count=0 > > > > > > > > > > > > > > > On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <bac...@li...> wrote: > > > > > > > > > > > Hi Martin, > > > > > > > > > > > > Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast: > > > > > > > Using Debug on bacula-dir, I get this: > > > > > > > > > > > > > > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25) > > > > > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan > > > > > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > > > > > > > > > Definitely deserves a thorough investigation. It's unlilkely to be > > > > > > caused by configuration. > > > > > > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958] > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/ > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked... > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback > > > > > > > > > > > > That file will now be relevant. > > > > > > > > > > > > /var/spool/bacula/bacula.2464062.traceback should contain information > > > > > > the developers can work with. > > > > > > > > > > > > Also interesting will be to know where you installed the packages from, > > > > > > or how you built the software. > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Arno > > > > > > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'. > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1. > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director. > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director. > > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected > > > > > > > > > > > > > > > > > > > > > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > > > > > > > > > > >> Also.. on the server running bacula/bacularis I get: > > > > > > >> > > > > > > >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000] > > > > > > >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping. > > > > > > >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000] > > > > > > >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0) > > > > > > >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00 > > > > > > >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000] > > > > > > >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0) > > > > > > >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > > > > >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000] > > > > > > >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000] > > > > > > >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping. > > > > > > >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000] > > > > > > >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000] > > > > > > >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0) > > > > > > >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > > > > >> > > > > > > >> > > > > > > >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > > > >> > > > > > > >>> Oh, I can see that.. > > > > > > >>> > > > > > > >>> The storage daemon says: > > > > > > >>> > > > > > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. > > > > > > >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > > > >>> > > > > > > >>> Config: > > > > > > >>> > > > > > > >>> Director { > > > > > > >>> Name = "bacula-dir" > > > > > > >>> Password = "@@SD_PASSWORD@@" > > > > > > >>> } > > > > > > >>> Director { > > > > > > >>> Name = "bacula-mon" > > > > > > >>> Password = "@@MON_SD_PASSWORD@@" > > > > > > >>> Monitor = yes > > > > > > >>> } > > > > > > >>> Storage { > > > > > > >>> Name = "bacula-sd" > > > > > > >>> WorkingDirectory = "/var/spool/bacula" > > > > > > >>> PidDirectory = "/var/run" > > > > > > >>> PluginDirectory = "/usr/lib64/bacula" > > > > > > >>> MaximumConcurrentJobs = 20 > > > > > > >>> } > > > > > > >>> Device { > > > > > > >>> Name = "AlwaysIncrement" > > > > > > >>> Description = "" > > > > > > >>> MediaType = "AlwaysIncrement" > > > > > > >>> DeviceType = "File" > > > > > > >>> ArchiveDevice = "/home/bacula/autoincrement" > > > > > > >>> RemovableMedia = no > > > > > > >>> RandomAccess = yes > > > > > > >>> AutomaticMount = yes > > > > > > >>> LabelMedia = yes > > > > > > >>> Autochanger = no > > > > > > >>> ReadOnly = no > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > >>> DriveIndex = 0 > > > > > > >>> } > > > > > > >>> Device { > > > > > > >>> Name = "FileChgr1-Dev1" > > > > > > >>> MediaType = "File1" > > > > > > >>> ArchiveDevice = "/tmp" > > > > > > >>> RemovableMedia = no > > > > > > >>> RandomAccess = yes > > > > > > >>> AutomaticMount = yes > > > > > > >>> LabelMedia = yes > > > > > > >>> AlwaysOpen = no > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > >>> } > > > > > > >>> Device { > > > > > > >>> Name = "FileChgr1-Dev2" > > > > > > >>> MediaType = "File1" > > > > > > >>> ArchiveDevice = "/tmp" > > > > > > >>> RemovableMedia = no > > > > > > >>> RandomAccess = yes > > > > > > >>> AutomaticMount = yes > > > > > > >>> LabelMedia = yes > > > > > > >>> AlwaysOpen = no > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > >>> } > > > > > > >>> Device { > > > > > > >>> Name = "FileChgr2-Dev1" > > > > > > >>> MediaType = "File2" > > > > > > >>> ArchiveDevice = "/tmp" > > > > > > >>> RemovableMedia = no > > > > > > >>> RandomAccess = yes > > > > > > >>> AutomaticMount = yes > > > > > > >>> LabelMedia = yes > > > > > > >>> AlwaysOpen = no > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > >>> } > > > > > > >>> Device { > > > > > > >>> Name = "FileChgr2-Dev2" > > > > > > >>> MediaType = "File2" > > > > > > >>> ArchiveDevice = "/tmp" > > > > > > >>> RemovableMedia = no > > > > > > >>> RandomAccess = yes > > > > > > >>> AutomaticMount = yes > > > > > > >>> LabelMedia = yes > > > > > > >>> AlwaysOpen = no > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > >>> } > > > > > > >>> Messages { > > > > > > >>> Name = "Standard" > > > > > > >>> Director = bacula-dir = All > > > > > > >>> } > > > > > > >>> Autochanger { > > > > > > >>> Name = "FileChgr1" > > > > > > >>> Device = "FileChgr1-Dev1" > > > > > > >>> Device = "FileChgr1-Dev2" > > > > > > >>> ChangerDevice = "/dev/null" > > > > > > >>> ChangerCommand = "" > > > > > > >>> } > > > > > > >>> Autochanger { > > > > > > >>> Name = "FileChgr2" > > > > > > >>> Device = "FileChgr2-Dev1" > > > > > > >>> Device = "FileChgr2-Dev2" > > > > > > >>> ChangerDevice = "/dev/null" > > > > > > >>> ChangerCommand = "" > > > > > > >>> } > > > > > > >>> Device { > > > > > > >>> DeviceType = "File" > > > > > > >>> RemovableMedia = no > > > > > > >>> AutomaticMount = yes > > > > > > >>> LabelMedia = yes > > > > > > >>> MaximumConcurrentJobs = 5 > > > > > > >>> RandomAccess = yes > > > > > > >>> Name = "Consolidate" > > > > > > >>> Description = "" > > > > > > >>> DriveIndex = 0 > > > > > > >>> ArchiveDevice = "/home/bacula/consolidate" > > > > > > >>> MediaType = "Consolidate" > > > > > > >>> ReadOnly = no > > > > > > >>> Autochanger = no > > > > > > >>> } > > > > > > >>> > > > > > > >>> Please say if I need to provide more configuration > > > > > > >>> > > > > > > >>> /Martin > > > > > > >>> > > > > > > >>> > > > > > > >>> Martin, > > > > > > >>> > > > > > > >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says". > > > > > > >>> > > > > > > >>> Regards, > > > > > > >>> Robert Gerber > > > > > > >>> 402-237-8692 > > > > > > >>> ro...@cr... > > > > > > >>> > > > > > > >>> > > > > > > >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: > > > > > > >>> > > > > > > >>> Hi guys > > > > > > >>> > > > > > > >>> Hope someone can help me.. > > > > > > >>> > > > > > > >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. > > > > > > >>> > > > > > > >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. > > > > > > >>> > > > > > > >>> On the hosts I get: > > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. > > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer > > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> The Storage daemon says > > > > > > >>> > > > > > > >>> > > > > > > >>> _______________________________________________ > > > > > > >>> Bacula-users mailing list > > > > > > >>> Bac...@li... > > > > > > >>> https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Bacula-users mailing list > > > > > > > Bac...@li... > > > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > -- > > > > > > Arno Lehmann > > > > > > > > > > > > IT-Service Lehmann > > > > > > Sandstr. 6, 49080 Osnabrück > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Bacula-users mailing list > > > > > > Bac...@li... > > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Bacula-users mailing list > > > > > Bac...@li... > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > |
|
From: <kjo...@ec...> - 2025-12-23 19:50:39
|
Hi, Again, thanks for the interest in my problem. I was certainly not expecting a patch for this old version of Bacula. This system will get a Bacula version update when the update to Debian occurs, according to the local policy governing this system. I actually did not assume that there was a Bacula bug here, though I could argue that there is a bug of not reporting more detailed error information from Postgresql -- I am sure that Postgresql provides more detailed information than 'Update failed'. It's too late to run the SQL command you suggested. The dbcheck run mentioned in my original post would have deleted it if it was present. I have not seen this error recur. If it does, I will look into the tracing you mention. While I have no direct experience with Bacula trace files, it does not seem that it would be impossible to use tools to find a database error, and then perhaps see the error returned by Postgres. I could be wrong about that. I might also be mistaken in my thinking that knowing why the update failed would be helpful. Best regards, Ken -----Original Message----- From: bac...@li... [mailto:bac...@li...] Sent: Thursday, December 11, 2025 2:13 PM To: bac...@li... Subject: Re: [Bacula-users] Troubleshoot bdb.h fatal error? Hi Ken, Am 11.12.2025 um 20:26 schrieb kjo...@ec...: > Rob, Arno, > > Thank you for taking an interest in my problem. You're welcome! Looks like the simple, obvious things do not help us here. So... > Answers to questions, as best as I can provide: > >> from Rob: >> You mentioned that the last two admin jobs failed. Was that a typo? If not, what errors did the last job (unmount, eject) give? > > The errors for jobid 27943 look very much like the errors for 27941. > > 08-Dec 14:21 linux2-dir JobId 27943: Fatal error: bdb.h:140 Update failed: affected_rows=0 for UPDATE Job SET JobStatus='R',Level=' ',StartTime='2025-12-08 14:21:57',ClientId=1,JobTDate=1765225317,PoolId=0,FileSetId=0 WHERE JobId=27943 > 08-Dec 14:21 linux2-dir JobId 27943: Fatal error: bdb.h:140 Update failed: affected_rows=0 for UPDATE Job SET JobStatus='f',Level=' ',StartTime='2025-12-08 14:21:57',ClientId=1,JobTDate=1765225317,PoolId=0,FileSetId=0 WHERE JobId=27943 We#ll need to find out what failed here. There is a simple possibility for the catalog update to fail, that is when the row its supposed to update does not exist. In bconsole, do sql select * from job where jobid=27943; and see if it finds that row. If it doesn't, I'm wondering why the fact that such a job could not be created was not reported -- it should have been. > 08-Dec 14:21 linux2-dir JobId 27943: Warning: Error updating job record. bdb.h:140 Update failed: affected_rows=0 for UPDATE Job SET JobStatus='f',EndTime='2025-12-08 14:21:57',ClientId=1,JobBytes=0,ReadBytes=0,JobFiles=0,JobErrors=1,VolSessionId=0,VolSessionTime=0,PoolId=0,FileSetId=0,JobTDate=1765225317,RealEndTime='2025-12-08 14:21:57',PriorJobId=0,HasBase=0,PurgedFiles=0 WHERE JobId=27943 > 08-Dec 14:21 linux2-dir JobId 27943: Warning: Error getting Job record for Job report: ERR=sql_get.c:303 No Job found for JobId 27943 We can probably guess the result of above exercise, but let's not guess :-) > 08-Dec 14:21 linux2-dir JobId 27943: Error: Bacula 9.6.7 (10Dec20): 08-Dec-2025 14:21:57 So we would have to investigate if the DIR for some reason "forgot" to create a job record when the job was started (I have never experienced such a thing, but that doesn't prove anything), if it didn't log it for some reason, if you just missed the error message (that would be convenient in this case :-) or if something deleted it in between successful job creation and the first update. Debugging, as a user, something that did *not* happen is a bit of a challenge, but we can probably achieve something if you can reproduce the problem. However, we'll probably not be able to convince Eric and team to fix issues in version 9 anymore. Thus -- would you be able to upgrade to a recent version, preferrbla the most recent one? I would recommend using the packages you can subscribe to at https://www.bacula.org/bacula-binary-package-download/ but, if that's not a choice you would consider, building from source is also an option. Proper packaging is above my pay grade, though :-) The alternative to enable tracing, debug, reproduce and eventually carefully read a few million lines of traces files will probably get us somewhere, but will not actually solve anything... Cheers, Arno -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück _______________________________________________ Bacula-users mailing list Bac...@li... https://lists.sourceforge.net/lists/listinfo/bacula-users |
|
From: Martin J. P. <m...@rt...> - 2025-12-20 00:03:28
|
Ok.. I finally get a real traceback: https://pastebin.com/sz7uWiYM Hope someone wiser than me can make some sense of it.. /Martin On Fredag, December 19, 2025 21:04 CET, Martin Simmons <ma...@li...> wrote: > The error: > > 'fail_time' has unknown type; cast it to its declared type > > means that gdb can't find any symbolic debugging information for bacula-dir > and/or libbac. If you installed bacula from rpm packages, then that > information is probably stripped out. > > Did the bacula compilation generate any debuginfo packages? If so, try > installing those as well. > > __Martin > > > >>>>> On Fri, 19 Dec 2025 20:20:42 +0100, Martin Juhl Prendergast said: > > > > Hi > > > > I'm not sure that I got gdb to work???: > > > > Check the log files for more information. > > > > [New LWP 3745058] > > [New LWP 3745057] > > [New LWP 3745056] > > [New LWP 3745055] > > [New LWP 3745054] > > [New LWP 3745053] > > [New LWP 3745052] > > [New LWP 3745051] > > [New LWP 3745034] > > [New LWP 3745033] > > [New LWP 3745032] > > [New LWP 3745031] > > [New LWP 3745030] > > [New LWP 3745029] > > [New LWP 3745025] > > [New LWP 3745024] > > [New LWP 3744994] > > [New LWP 3744993] > > [New LWP 3744988] > > [New LWP 3744981] > > [New LWP 3069995] > > [New LWP 3069994] > > [New LWP 3069985] > > [New LWP 3069968] > > [New LWP 3069967] > > [New LWP 3069965] > > [New LWP 3069963] > > [New LWP 3069962] > > [New LWP 3060166] > > [New LWP 3060113] > > [New LWP 2985337] > > [New LWP 2985336] > > [New LWP 2985335] > > [New LWP 2985332] > > [Thread debugging using libthread_db enabled] > > Using host libthread_db library "/lib64/libthread_db.so.1". > > 0x00007f968dc8837a in __futex_abstimed_wait_common () from /lib64/libc.so.6 > > /usr/libexec/bacula/btraceback.gdb:1: Error in sourced command file: > > 'fail_time' has unknown type; cast it to its declared type > > [Inferior 1 (process 2985331) detached] > > Attempt to dump locks > > threadid=0x7f9606ffd640 max=1 current=-1 > > threadid=0x7f9627fff640 max=1 current=-1 > > threadid=0x7f9684ff9640 max=1 current=-1 > > threadid=0x7f96857fa640 max=1 current=-1 > > threadid=0x7f9666ffd640 max=1 current=-1 > > threadid=0x7f9667fff640 max=1 current=-1 > > threadid=0x7f9664ff9640 max=1 current=-1 > > threadid=0x7f9605ffb640 max=1 current=-1 > > threadid=0x7f96277fe640 max=1 current=-1 > > threadid=0x7f96477fe640 max=1 current=-1 > > threadid=0x7f96657fa640 max=1 current=-1 > > threadid=0x7f9646ffd640 max=1 current=-1 > > threadid=0x7f96077fe640 max=1 current=-1 > > threadid=0x7f96467fc640 max=1 current=-1 > > threadid=0x7f96677fe640 max=1 current=-1 > > threadid=0x7f96067fc640 max=1 current=-1 > > threadid=0x7f9607fff640 max=2 current=-1 > > threadid=0x7f9645ffb640 max=1 current=-1 > > threadid=0x7f9647fff640 max=1 current=-1 > > threadid=0x7f9665ffb640 max=1 current=-1 > > threadid=0x7f9624ff9640 max=2 current=-1 > > threadid=0x7f96257fa640 max=2 current=-1 > > threadid=0x7f9625ffb640 max=2 current=-1 > > threadid=0x7f9685ffb640 max=2 current=-1 > > threadid=0x7f9644ff9640 max=2 current=-1 > > threadid=0x7f96457fa640 max=2 current=-1 > > threadid=0x7f96867fc640 max=2 current=-1 > > threadid=0x7f96667fc640 max=2 current=-1 > > threadid=0x7f96267fc640 max=2 current=-1 > > threadid=0x7f9626ffd640 max=2 current=-1 > > threadid=0x7f9686ffd640 max=1 current=-1 > > threadid=0x7f96877fe640 max=2 current=-1 > > threadid=0x7f9687fff640 max=0 current=-1 > > threadid=0x7f968cfaa640 max=0 current=-1 > > threadid=0x7f968de12f40 max=1 current=-1 > > Attempt to dump current JCRs. njcrs=7 > > threadid=0x7f968de12f40 JobId=0 JobStatus=R jcr=0x55eae96c6da8 name=*JobMonitor*.2025-12-18_22.57.48_01 > > use_count=1 killable=0 > > JobType=I JobLevel= > > sched_time=18-Dec-2025 22:57 start_time=18-Dec-2025 22:57 > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > db=(nil) db_batch=(nil) batch_started=0 > > wstore=0x55eae95f64e8 rstore=(nil) wjcr=(nil) client=0x55eae95f3788 reschedule_count=0 SD_msg_chan_started=0 > > threadid=0x7f9626ffd640 JobId=61 JobStatus=R jcr=0x7f963c015e88 name=SullustBackup.2025-12-19_00.53.46_20 > > use_count=2 killable=1 > > JobType=B JobLevel=F > > sched_time=19-Dec-2025 00:53 start_time=19-Dec-2025 00:53 > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > db=0x7f963c024e88 db_batch=(nil) batch_started=0 > > wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=1 > > BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384 > > RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0 > > threadid=0x7f9686ffd640 JobId=67 JobStatus=c jcr=0x7f967c01c908 name=SullustBackup.2025-12-19_01.00.00_17 > > use_count=1 killable=0 > > JobType=B JobLevel=F > > sched_time=19-Dec-2025 01:00 start_time=19-Dec-2025 01:00 > > end_time=01-Jan-1970 01:00 wait_time=19-Dec-2025 01:00 > > db=0x7f963c024e88 db_batch=(nil) batch_started=0 > > wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=0 > > BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384 > > RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0 > > threadid=0x7f9607fff640 JobId=0 JobStatus=R jcr=0x7f962800b328 name=-Console-.2025-12-19_20.11.56_53 > > use_count=1 killable=0 > > JobType=U JobLevel= > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > db=(nil) db_batch=(nil) batch_started=0 > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > threadid=0x7f9665ffb640 JobId=0 JobStatus=R jcr=0x7f962c00b6f8 name=-Console-.2025-12-19_20.11.57_20 > > use_count=1 killable=0 > > JobType=U JobLevel= > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > db=(nil) db_batch=(nil) batch_started=0 > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > threadid=0x7f9606ffd640 JobId=0 JobStatus=R jcr=0x7f963c00f5f8 name=-Console-.2025-12-19_20.11.58_37 > > use_count=1 killable=0 > > JobType=U JobLevel= > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > db=(nil) db_batch=(nil) batch_started=0 > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > threadid=0x7f9627fff640 JobId=0 JobStatus=R jcr=0x7f963800db68 name=-Console-.2025-12-19_20.11.58_38 > > use_count=1 killable=0 > > JobType=U JobLevel= > > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > > db=(nil) db_batch=(nil) batch_started=0 > > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > > List plugins. Hook count=0 > > > > On Torsdag, December 18, 2025 15:38 CET, Martin Simmons <ma...@li...> wrote: > > > > > You will need to install gdb as well, so it can get backtraces. > > > > > > __Martin > > > > > > > > > >>>>> On Thu, 18 Dec 2025 14:41:46 +0100, Martin Juhl Prendergast said: > > > > > > > > Hi Arno > > > > > > > > Traceback is inserted below.. > > > > > > > > Original I installed the packages from the EPEL9 repository, but when I had the problem there, I rebuilt the 15.0.3 package from Fedora 44, on RHEL9... only to see the same issue.. > > > > > > > > I have started the storage daemon in debug mode, and is waiting to see debug for that, the next time it crashes.. > > > > > > > > Regards > > > > > > > > If you need any more info, please > > > > > > > > Check the log files for more information. > > > > > > > > Please install a debugger (gdb) to receive a traceback. > > > > Attempt to dump locks > > > > threadid=0x7fc5477fe640 max=1 current=-1 > > > > threadid=0x7fc564ff9640 max=1 current=-1 > > > > threadid=0x7fc5467fc640 max=1 current=-1 > > > > threadid=0x7fc547fff640 max=2 current=-1 > > > > threadid=0x7fc5457fa640 max=1 current=-1 > > > > threadid=0x7fc545ffb640 max=2 current=-1 > > > > threadid=0x7fc59d80e640 max=1 current=-1 > > > > threadid=0x7fc59e00f640 max=2 current=-1 > > > > threadid=0x7fc59e810640 max=0 current=-1 > > > > threadid=0x7fc59f1ff640 max=0 current=-1 > > > > threadid=0x7fc5a000bf40 max=1 current=-1 > > > > Attempt to dump current JCRs. njcrs=3 > > > > threadid=0x7fc5a000bf40 JobId=0 JobStatus=R jcr=0x558b1d983008 name=*JobMonitor*.2025-12-18_11.30.53_01 > > > > use_count=1 killable=0 > > > > JobType=I JobLevel= > > > > sched_time=18-Dec-2025 11:30 start_time=18-Dec-2025 11:30 > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > > > > threadid=0x7fc545ffb640 JobId=54 JobStatus=R jcr=0x7fc5700187b8 name=SullustBackup.2025-12-18_11.35.50_45 > > > > use_count=2 killable=1 > > > > JobType=B JobLevel=F > > > > sched_time=18-Dec-2025 11:35 start_time=18-Dec-2025 11:35 > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > db=0x7fc570027348 db_batch=(nil) batch_started=0 > > > > wstore=0x558b1d8afab8 rstore=(nil) wjcr=(nil) client=0x558b1d8ad5e8 reschedule_count=0 SD_msg_chan_started=1 > > > > BDB=0x7fc570027348 db_name=bacula db_user=bacula connected=true > > > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=16 > > > > RWLOCK=0x7fc570027360 w_active=0 w_wait=0 > > > > threadid=0x7fc547fff640 JobId=0 JobStatus=R jcr=0x7fc53400b098 name=-Console-.2025-12-18_11.36.53_00 > > > > use_count=1 killable=0 > > > > JobType=U JobLevel= > > > > sched_time=18-Dec-2025 11:36 start_time=18-Dec-2025 11:36 > > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > > db=(nil) db_batch=(nil) batch_started=0 > > > > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > > > > List plugins. Hook count=0 > > > > > > > > > > > > On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <bac...@li...> wrote: > > > > > > > > > Hi Martin, > > > > > > > > > > Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast: > > > > > > Using Debug on bacula-dir, I get this: > > > > > > > > > > > > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25) > > > > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan > > > > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > > > > > > > Definitely deserves a thorough investigation. It's unlilkely to be > > > > > caused by configuration. > > > > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958] > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/ > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked... > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback > > > > > > > > > > That file will now be relevant. > > > > > > > > > > /var/spool/bacula/bacula.2464062.traceback should contain information > > > > > the developers can work with. > > > > > > > > > > Also interesting will be to know where you installed the packages from, > > > > > or how you built the software. > > > > > > > > > > Cheers, > > > > > > > > > > Arno > > > > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'. > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1. > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director. > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director. > > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected > > > > > > > > > > > > > > > > > > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > > > > > > > > >> Also.. on the server running bacula/bacularis I get: > > > > > >> > > > > > >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000] > > > > > >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping. > > > > > >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000] > > > > > >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0) > > > > > >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00 > > > > > >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000] > > > > > >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0) > > > > > >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > > > >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000] > > > > > >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000] > > > > > >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping. > > > > > >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000] > > > > > >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000] > > > > > >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0) > > > > > >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > > > >> > > > > > >> > > > > > >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > > >> > > > > > >>> Oh, I can see that.. > > > > > >>> > > > > > >>> The storage daemon says: > > > > > >>> > > > > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. > > > > > >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > > >>> > > > > > >>> Config: > > > > > >>> > > > > > >>> Director { > > > > > >>> Name = "bacula-dir" > > > > > >>> Password = "@@SD_PASSWORD@@" > > > > > >>> } > > > > > >>> Director { > > > > > >>> Name = "bacula-mon" > > > > > >>> Password = "@@MON_SD_PASSWORD@@" > > > > > >>> Monitor = yes > > > > > >>> } > > > > > >>> Storage { > > > > > >>> Name = "bacula-sd" > > > > > >>> WorkingDirectory = "/var/spool/bacula" > > > > > >>> PidDirectory = "/var/run" > > > > > >>> PluginDirectory = "/usr/lib64/bacula" > > > > > >>> MaximumConcurrentJobs = 20 > > > > > >>> } > > > > > >>> Device { > > > > > >>> Name = "AlwaysIncrement" > > > > > >>> Description = "" > > > > > >>> MediaType = "AlwaysIncrement" > > > > > >>> DeviceType = "File" > > > > > >>> ArchiveDevice = "/home/bacula/autoincrement" > > > > > >>> RemovableMedia = no > > > > > >>> RandomAccess = yes > > > > > >>> AutomaticMount = yes > > > > > >>> LabelMedia = yes > > > > > >>> Autochanger = no > > > > > >>> ReadOnly = no > > > > > >>> MaximumConcurrentJobs = 5 > > > > > >>> DriveIndex = 0 > > > > > >>> } > > > > > >>> Device { > > > > > >>> Name = "FileChgr1-Dev1" > > > > > >>> MediaType = "File1" > > > > > >>> ArchiveDevice = "/tmp" > > > > > >>> RemovableMedia = no > > > > > >>> RandomAccess = yes > > > > > >>> AutomaticMount = yes > > > > > >>> LabelMedia = yes > > > > > >>> AlwaysOpen = no > > > > > >>> MaximumConcurrentJobs = 5 > > > > > >>> } > > > > > >>> Device { > > > > > >>> Name = "FileChgr1-Dev2" > > > > > >>> MediaType = "File1" > > > > > >>> ArchiveDevice = "/tmp" > > > > > >>> RemovableMedia = no > > > > > >>> RandomAccess = yes > > > > > >>> AutomaticMount = yes > > > > > >>> LabelMedia = yes > > > > > >>> AlwaysOpen = no > > > > > >>> MaximumConcurrentJobs = 5 > > > > > >>> } > > > > > >>> Device { > > > > > >>> Name = "FileChgr2-Dev1" > > > > > >>> MediaType = "File2" > > > > > >>> ArchiveDevice = "/tmp" > > > > > >>> RemovableMedia = no > > > > > >>> RandomAccess = yes > > > > > >>> AutomaticMount = yes > > > > > >>> LabelMedia = yes > > > > > >>> AlwaysOpen = no > > > > > >>> MaximumConcurrentJobs = 5 > > > > > >>> } > > > > > >>> Device { > > > > > >>> Name = "FileChgr2-Dev2" > > > > > >>> MediaType = "File2" > > > > > >>> ArchiveDevice = "/tmp" > > > > > >>> RemovableMedia = no > > > > > >>> RandomAccess = yes > > > > > >>> AutomaticMount = yes > > > > > >>> LabelMedia = yes > > > > > >>> AlwaysOpen = no > > > > > >>> MaximumConcurrentJobs = 5 > > > > > >>> } > > > > > >>> Messages { > > > > > >>> Name = "Standard" > > > > > >>> Director = bacula-dir = All > > > > > >>> } > > > > > >>> Autochanger { > > > > > >>> Name = "FileChgr1" > > > > > >>> Device = "FileChgr1-Dev1" > > > > > >>> Device = "FileChgr1-Dev2" > > > > > >>> ChangerDevice = "/dev/null" > > > > > >>> ChangerCommand = "" > > > > > >>> } > > > > > >>> Autochanger { > > > > > >>> Name = "FileChgr2" > > > > > >>> Device = "FileChgr2-Dev1" > > > > > >>> Device = "FileChgr2-Dev2" > > > > > >>> ChangerDevice = "/dev/null" > > > > > >>> ChangerCommand = "" > > > > > >>> } > > > > > >>> Device { > > > > > >>> DeviceType = "File" > > > > > >>> RemovableMedia = no > > > > > >>> AutomaticMount = yes > > > > > >>> LabelMedia = yes > > > > > >>> MaximumConcurrentJobs = 5 > > > > > >>> RandomAccess = yes > > > > > >>> Name = "Consolidate" > > > > > >>> Description = "" > > > > > >>> DriveIndex = 0 > > > > > >>> ArchiveDevice = "/home/bacula/consolidate" > > > > > >>> MediaType = "Consolidate" > > > > > >>> ReadOnly = no > > > > > >>> Autochanger = no > > > > > >>> } > > > > > >>> > > > > > >>> Please say if I need to provide more configuration > > > > > >>> > > > > > >>> /Martin > > > > > >>> > > > > > >>> > > > > > >>> Martin, > > > > > >>> > > > > > >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says". > > > > > >>> > > > > > >>> Regards, > > > > > >>> Robert Gerber > > > > > >>> 402-237-8692 > > > > > >>> ro...@cr... > > > > > >>> > > > > > >>> > > > > > >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: > > > > > >>> > > > > > >>> Hi guys > > > > > >>> > > > > > >>> Hope someone can help me.. > > > > > >>> > > > > > >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. > > > > > >>> > > > > > >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. > > > > > >>> > > > > > >>> On the hosts I get: > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer > > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> The Storage daemon says > > > > > >>> > > > > > >>> > > > > > >>> _______________________________________________ > > > > > >>> Bacula-users mailing list > > > > > >>> Bac...@li... > > > > > >>> https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Bacula-users mailing list > > > > > > Bac...@li... > > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > -- > > > > > Arno Lehmann > > > > > > > > > > IT-Service Lehmann > > > > > Sandstr. 6, 49080 Osnabrück > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Bacula-users mailing list > > > > > Bac...@li... > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > > > _______________________________________________ > > > > Bacula-users mailing list > > > > Bac...@li... > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > |
|
From: Martin S. <ma...@li...> - 2025-12-19 20:04:44
|
The error: 'fail_time' has unknown type; cast it to its declared type means that gdb can't find any symbolic debugging information for bacula-dir and/or libbac. If you installed bacula from rpm packages, then that information is probably stripped out. Did the bacula compilation generate any debuginfo packages? If so, try installing those as well. __Martin >>>>> On Fri, 19 Dec 2025 20:20:42 +0100, Martin Juhl Prendergast said: > > Hi > > I'm not sure that I got gdb to work???: > > Check the log files for more information. > > [New LWP 3745058] > [New LWP 3745057] > [New LWP 3745056] > [New LWP 3745055] > [New LWP 3745054] > [New LWP 3745053] > [New LWP 3745052] > [New LWP 3745051] > [New LWP 3745034] > [New LWP 3745033] > [New LWP 3745032] > [New LWP 3745031] > [New LWP 3745030] > [New LWP 3745029] > [New LWP 3745025] > [New LWP 3745024] > [New LWP 3744994] > [New LWP 3744993] > [New LWP 3744988] > [New LWP 3744981] > [New LWP 3069995] > [New LWP 3069994] > [New LWP 3069985] > [New LWP 3069968] > [New LWP 3069967] > [New LWP 3069965] > [New LWP 3069963] > [New LWP 3069962] > [New LWP 3060166] > [New LWP 3060113] > [New LWP 2985337] > [New LWP 2985336] > [New LWP 2985335] > [New LWP 2985332] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > 0x00007f968dc8837a in __futex_abstimed_wait_common () from /lib64/libc.so.6 > /usr/libexec/bacula/btraceback.gdb:1: Error in sourced command file: > 'fail_time' has unknown type; cast it to its declared type > [Inferior 1 (process 2985331) detached] > Attempt to dump locks > threadid=0x7f9606ffd640 max=1 current=-1 > threadid=0x7f9627fff640 max=1 current=-1 > threadid=0x7f9684ff9640 max=1 current=-1 > threadid=0x7f96857fa640 max=1 current=-1 > threadid=0x7f9666ffd640 max=1 current=-1 > threadid=0x7f9667fff640 max=1 current=-1 > threadid=0x7f9664ff9640 max=1 current=-1 > threadid=0x7f9605ffb640 max=1 current=-1 > threadid=0x7f96277fe640 max=1 current=-1 > threadid=0x7f96477fe640 max=1 current=-1 > threadid=0x7f96657fa640 max=1 current=-1 > threadid=0x7f9646ffd640 max=1 current=-1 > threadid=0x7f96077fe640 max=1 current=-1 > threadid=0x7f96467fc640 max=1 current=-1 > threadid=0x7f96677fe640 max=1 current=-1 > threadid=0x7f96067fc640 max=1 current=-1 > threadid=0x7f9607fff640 max=2 current=-1 > threadid=0x7f9645ffb640 max=1 current=-1 > threadid=0x7f9647fff640 max=1 current=-1 > threadid=0x7f9665ffb640 max=1 current=-1 > threadid=0x7f9624ff9640 max=2 current=-1 > threadid=0x7f96257fa640 max=2 current=-1 > threadid=0x7f9625ffb640 max=2 current=-1 > threadid=0x7f9685ffb640 max=2 current=-1 > threadid=0x7f9644ff9640 max=2 current=-1 > threadid=0x7f96457fa640 max=2 current=-1 > threadid=0x7f96867fc640 max=2 current=-1 > threadid=0x7f96667fc640 max=2 current=-1 > threadid=0x7f96267fc640 max=2 current=-1 > threadid=0x7f9626ffd640 max=2 current=-1 > threadid=0x7f9686ffd640 max=1 current=-1 > threadid=0x7f96877fe640 max=2 current=-1 > threadid=0x7f9687fff640 max=0 current=-1 > threadid=0x7f968cfaa640 max=0 current=-1 > threadid=0x7f968de12f40 max=1 current=-1 > Attempt to dump current JCRs. njcrs=7 > threadid=0x7f968de12f40 JobId=0 JobStatus=R jcr=0x55eae96c6da8 name=*JobMonitor*.2025-12-18_22.57.48_01 > use_count=1 killable=0 > JobType=I JobLevel= > sched_time=18-Dec-2025 22:57 start_time=18-Dec-2025 22:57 > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x55eae95f64e8 rstore=(nil) wjcr=(nil) client=0x55eae95f3788 reschedule_count=0 SD_msg_chan_started=0 > threadid=0x7f9626ffd640 JobId=61 JobStatus=R jcr=0x7f963c015e88 name=SullustBackup.2025-12-19_00.53.46_20 > use_count=2 killable=1 > JobType=B JobLevel=F > sched_time=19-Dec-2025 00:53 start_time=19-Dec-2025 00:53 > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > db=0x7f963c024e88 db_batch=(nil) batch_started=0 > wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=1 > BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384 > RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0 > threadid=0x7f9686ffd640 JobId=67 JobStatus=c jcr=0x7f967c01c908 name=SullustBackup.2025-12-19_01.00.00_17 > use_count=1 killable=0 > JobType=B JobLevel=F > sched_time=19-Dec-2025 01:00 start_time=19-Dec-2025 01:00 > end_time=01-Jan-1970 01:00 wait_time=19-Dec-2025 01:00 > db=0x7f963c024e88 db_batch=(nil) batch_started=0 > wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=0 > BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384 > RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0 > threadid=0x7f9607fff640 JobId=0 JobStatus=R jcr=0x7f962800b328 name=-Console-.2025-12-19_20.11.56_53 > use_count=1 killable=0 > JobType=U JobLevel= > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > threadid=0x7f9665ffb640 JobId=0 JobStatus=R jcr=0x7f962c00b6f8 name=-Console-.2025-12-19_20.11.57_20 > use_count=1 killable=0 > JobType=U JobLevel= > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > threadid=0x7f9606ffd640 JobId=0 JobStatus=R jcr=0x7f963c00f5f8 name=-Console-.2025-12-19_20.11.58_37 > use_count=1 killable=0 > JobType=U JobLevel= > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > threadid=0x7f9627fff640 JobId=0 JobStatus=R jcr=0x7f963800db68 name=-Console-.2025-12-19_20.11.58_38 > use_count=1 killable=0 > JobType=U JobLevel= > sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11 > end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0 > List plugins. Hook count=0 > > On Torsdag, December 18, 2025 15:38 CET, Martin Simmons <ma...@li...> wrote: > > > You will need to install gdb as well, so it can get backtraces. > > > > __Martin > > > > > > >>>>> On Thu, 18 Dec 2025 14:41:46 +0100, Martin Juhl Prendergast said: > > > > > > Hi Arno > > > > > > Traceback is inserted below.. > > > > > > Original I installed the packages from the EPEL9 repository, but when I had the problem there, I rebuilt the 15.0.3 package from Fedora 44, on RHEL9... only to see the same issue.. > > > > > > I have started the storage daemon in debug mode, and is waiting to see debug for that, the next time it crashes.. > > > > > > Regards > > > > > > If you need any more info, please > > > > > > Check the log files for more information. > > > > > > Please install a debugger (gdb) to receive a traceback. > > > Attempt to dump locks > > > threadid=0x7fc5477fe640 max=1 current=-1 > > > threadid=0x7fc564ff9640 max=1 current=-1 > > > threadid=0x7fc5467fc640 max=1 current=-1 > > > threadid=0x7fc547fff640 max=2 current=-1 > > > threadid=0x7fc5457fa640 max=1 current=-1 > > > threadid=0x7fc545ffb640 max=2 current=-1 > > > threadid=0x7fc59d80e640 max=1 current=-1 > > > threadid=0x7fc59e00f640 max=2 current=-1 > > > threadid=0x7fc59e810640 max=0 current=-1 > > > threadid=0x7fc59f1ff640 max=0 current=-1 > > > threadid=0x7fc5a000bf40 max=1 current=-1 > > > Attempt to dump current JCRs. njcrs=3 > > > threadid=0x7fc5a000bf40 JobId=0 JobStatus=R jcr=0x558b1d983008 name=*JobMonitor*.2025-12-18_11.30.53_01 > > > use_count=1 killable=0 > > > JobType=I JobLevel= > > > sched_time=18-Dec-2025 11:30 start_time=18-Dec-2025 11:30 > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > db=(nil) db_batch=(nil) batch_started=0 > > > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > > > threadid=0x7fc545ffb640 JobId=54 JobStatus=R jcr=0x7fc5700187b8 name=SullustBackup.2025-12-18_11.35.50_45 > > > use_count=2 killable=1 > > > JobType=B JobLevel=F > > > sched_time=18-Dec-2025 11:35 start_time=18-Dec-2025 11:35 > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > db=0x7fc570027348 db_batch=(nil) batch_started=0 > > > wstore=0x558b1d8afab8 rstore=(nil) wjcr=(nil) client=0x558b1d8ad5e8 reschedule_count=0 SD_msg_chan_started=1 > > > BDB=0x7fc570027348 db_name=bacula db_user=bacula connected=true > > > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=16 > > > RWLOCK=0x7fc570027360 w_active=0 w_wait=0 > > > threadid=0x7fc547fff640 JobId=0 JobStatus=R jcr=0x7fc53400b098 name=-Console-.2025-12-18_11.36.53_00 > > > use_count=1 killable=0 > > > JobType=U JobLevel= > > > sched_time=18-Dec-2025 11:36 start_time=18-Dec-2025 11:36 > > > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > > > db=(nil) db_batch=(nil) batch_started=0 > > > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > > > List plugins. Hook count=0 > > > > > > > > > On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <bac...@li...> wrote: > > > > > > > Hi Martin, > > > > > > > > Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast: > > > > > Using Debug on bacula-dir, I get this: > > > > > > > > > > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25) > > > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan > > > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > > > > > Definitely deserves a thorough investigation. It's unlilkely to be > > > > caused by configuration. > > > > > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958] > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/ > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked... > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback > > > > > > > > That file will now be relevant. > > > > > > > > /var/spool/bacula/bacula.2464062.traceback should contain information > > > > the developers can work with. > > > > > > > > Also interesting will be to know where you installed the packages from, > > > > or how you built the software. > > > > > > > > Cheers, > > > > > > > > Arno > > > > > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'. > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1. > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director. > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director. > > > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected > > > > > > > > > > > > > > > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > > > > > > >> Also.. on the server running bacula/bacularis I get: > > > > >> > > > > >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000] > > > > >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping. > > > > >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000] > > > > >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0) > > > > >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00 > > > > >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000] > > > > >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0) > > > > >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > > >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000] > > > > >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000] > > > > >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping. > > > > >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000] > > > > >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000] > > > > >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0) > > > > >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > > >> > > > > >> > > > > >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > >> > > > > >>> Oh, I can see that.. > > > > >>> > > > > >>> The storage daemon says: > > > > >>> > > > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. > > > > >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > >>> > > > > >>> Config: > > > > >>> > > > > >>> Director { > > > > >>> Name = "bacula-dir" > > > > >>> Password = "@@SD_PASSWORD@@" > > > > >>> } > > > > >>> Director { > > > > >>> Name = "bacula-mon" > > > > >>> Password = "@@MON_SD_PASSWORD@@" > > > > >>> Monitor = yes > > > > >>> } > > > > >>> Storage { > > > > >>> Name = "bacula-sd" > > > > >>> WorkingDirectory = "/var/spool/bacula" > > > > >>> PidDirectory = "/var/run" > > > > >>> PluginDirectory = "/usr/lib64/bacula" > > > > >>> MaximumConcurrentJobs = 20 > > > > >>> } > > > > >>> Device { > > > > >>> Name = "AlwaysIncrement" > > > > >>> Description = "" > > > > >>> MediaType = "AlwaysIncrement" > > > > >>> DeviceType = "File" > > > > >>> ArchiveDevice = "/home/bacula/autoincrement" > > > > >>> RemovableMedia = no > > > > >>> RandomAccess = yes > > > > >>> AutomaticMount = yes > > > > >>> LabelMedia = yes > > > > >>> Autochanger = no > > > > >>> ReadOnly = no > > > > >>> MaximumConcurrentJobs = 5 > > > > >>> DriveIndex = 0 > > > > >>> } > > > > >>> Device { > > > > >>> Name = "FileChgr1-Dev1" > > > > >>> MediaType = "File1" > > > > >>> ArchiveDevice = "/tmp" > > > > >>> RemovableMedia = no > > > > >>> RandomAccess = yes > > > > >>> AutomaticMount = yes > > > > >>> LabelMedia = yes > > > > >>> AlwaysOpen = no > > > > >>> MaximumConcurrentJobs = 5 > > > > >>> } > > > > >>> Device { > > > > >>> Name = "FileChgr1-Dev2" > > > > >>> MediaType = "File1" > > > > >>> ArchiveDevice = "/tmp" > > > > >>> RemovableMedia = no > > > > >>> RandomAccess = yes > > > > >>> AutomaticMount = yes > > > > >>> LabelMedia = yes > > > > >>> AlwaysOpen = no > > > > >>> MaximumConcurrentJobs = 5 > > > > >>> } > > > > >>> Device { > > > > >>> Name = "FileChgr2-Dev1" > > > > >>> MediaType = "File2" > > > > >>> ArchiveDevice = "/tmp" > > > > >>> RemovableMedia = no > > > > >>> RandomAccess = yes > > > > >>> AutomaticMount = yes > > > > >>> LabelMedia = yes > > > > >>> AlwaysOpen = no > > > > >>> MaximumConcurrentJobs = 5 > > > > >>> } > > > > >>> Device { > > > > >>> Name = "FileChgr2-Dev2" > > > > >>> MediaType = "File2" > > > > >>> ArchiveDevice = "/tmp" > > > > >>> RemovableMedia = no > > > > >>> RandomAccess = yes > > > > >>> AutomaticMount = yes > > > > >>> LabelMedia = yes > > > > >>> AlwaysOpen = no > > > > >>> MaximumConcurrentJobs = 5 > > > > >>> } > > > > >>> Messages { > > > > >>> Name = "Standard" > > > > >>> Director = bacula-dir = All > > > > >>> } > > > > >>> Autochanger { > > > > >>> Name = "FileChgr1" > > > > >>> Device = "FileChgr1-Dev1" > > > > >>> Device = "FileChgr1-Dev2" > > > > >>> ChangerDevice = "/dev/null" > > > > >>> ChangerCommand = "" > > > > >>> } > > > > >>> Autochanger { > > > > >>> Name = "FileChgr2" > > > > >>> Device = "FileChgr2-Dev1" > > > > >>> Device = "FileChgr2-Dev2" > > > > >>> ChangerDevice = "/dev/null" > > > > >>> ChangerCommand = "" > > > > >>> } > > > > >>> Device { > > > > >>> DeviceType = "File" > > > > >>> RemovableMedia = no > > > > >>> AutomaticMount = yes > > > > >>> LabelMedia = yes > > > > >>> MaximumConcurrentJobs = 5 > > > > >>> RandomAccess = yes > > > > >>> Name = "Consolidate" > > > > >>> Description = "" > > > > >>> DriveIndex = 0 > > > > >>> ArchiveDevice = "/home/bacula/consolidate" > > > > >>> MediaType = "Consolidate" > > > > >>> ReadOnly = no > > > > >>> Autochanger = no > > > > >>> } > > > > >>> > > > > >>> Please say if I need to provide more configuration > > > > >>> > > > > >>> /Martin > > > > >>> > > > > >>> > > > > >>> Martin, > > > > >>> > > > > >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says". > > > > >>> > > > > >>> Regards, > > > > >>> Robert Gerber > > > > >>> 402-237-8692 > > > > >>> ro...@cr... > > > > >>> > > > > >>> > > > > >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: > > > > >>> > > > > >>> Hi guys > > > > >>> > > > > >>> Hope someone can help me.. > > > > >>> > > > > >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. > > > > >>> > > > > >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. > > > > >>> > > > > >>> On the hosts I get: > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer > > > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 > > > > >>> > > > > >>> > > > > >>> > > > > >>> The Storage daemon says > > > > >>> > > > > >>> > > > > >>> _______________________________________________ > > > > >>> Bacula-users mailing list > > > > >>> Bac...@li... > > > > >>> https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Bacula-users mailing list > > > > > Bac...@li... > > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > -- > > > > Arno Lehmann > > > > > > > > IT-Service Lehmann > > > > Sandstr. 6, 49080 Osnabrück > > > > > > > > > > > > > > > > _______________________________________________ > > > > Bacula-users mailing list > > > > Bac...@li... > > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > _______________________________________________ > > > Bacula-users mailing list > > > Bac...@li... > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > |
|
From: Martin J. P. <m...@rt...> - 2025-12-19 19:21:04
|
Hi
I'm not sure that I got gdb to work???:
Check the log files for more information.
[New LWP 3745058]
[New LWP 3745057]
[New LWP 3745056]
[New LWP 3745055]
[New LWP 3745054]
[New LWP 3745053]
[New LWP 3745052]
[New LWP 3745051]
[New LWP 3745034]
[New LWP 3745033]
[New LWP 3745032]
[New LWP 3745031]
[New LWP 3745030]
[New LWP 3745029]
[New LWP 3745025]
[New LWP 3745024]
[New LWP 3744994]
[New LWP 3744993]
[New LWP 3744988]
[New LWP 3744981]
[New LWP 3069995]
[New LWP 3069994]
[New LWP 3069985]
[New LWP 3069968]
[New LWP 3069967]
[New LWP 3069965]
[New LWP 3069963]
[New LWP 3069962]
[New LWP 3060166]
[New LWP 3060113]
[New LWP 2985337]
[New LWP 2985336]
[New LWP 2985335]
[New LWP 2985332]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f968dc8837a in __futex_abstimed_wait_common () from /lib64/libc.so.6
/usr/libexec/bacula/btraceback.gdb:1: Error in sourced command file:
'fail_time' has unknown type; cast it to its declared type
[Inferior 1 (process 2985331) detached]
Attempt to dump locks
threadid=0x7f9606ffd640 max=1 current=-1
threadid=0x7f9627fff640 max=1 current=-1
threadid=0x7f9684ff9640 max=1 current=-1
threadid=0x7f96857fa640 max=1 current=-1
threadid=0x7f9666ffd640 max=1 current=-1
threadid=0x7f9667fff640 max=1 current=-1
threadid=0x7f9664ff9640 max=1 current=-1
threadid=0x7f9605ffb640 max=1 current=-1
threadid=0x7f96277fe640 max=1 current=-1
threadid=0x7f96477fe640 max=1 current=-1
threadid=0x7f96657fa640 max=1 current=-1
threadid=0x7f9646ffd640 max=1 current=-1
threadid=0x7f96077fe640 max=1 current=-1
threadid=0x7f96467fc640 max=1 current=-1
threadid=0x7f96677fe640 max=1 current=-1
threadid=0x7f96067fc640 max=1 current=-1
threadid=0x7f9607fff640 max=2 current=-1
threadid=0x7f9645ffb640 max=1 current=-1
threadid=0x7f9647fff640 max=1 current=-1
threadid=0x7f9665ffb640 max=1 current=-1
threadid=0x7f9624ff9640 max=2 current=-1
threadid=0x7f96257fa640 max=2 current=-1
threadid=0x7f9625ffb640 max=2 current=-1
threadid=0x7f9685ffb640 max=2 current=-1
threadid=0x7f9644ff9640 max=2 current=-1
threadid=0x7f96457fa640 max=2 current=-1
threadid=0x7f96867fc640 max=2 current=-1
threadid=0x7f96667fc640 max=2 current=-1
threadid=0x7f96267fc640 max=2 current=-1
threadid=0x7f9626ffd640 max=2 current=-1
threadid=0x7f9686ffd640 max=1 current=-1
threadid=0x7f96877fe640 max=2 current=-1
threadid=0x7f9687fff640 max=0 current=-1
threadid=0x7f968cfaa640 max=0 current=-1
threadid=0x7f968de12f40 max=1 current=-1
Attempt to dump current JCRs. njcrs=7
threadid=0x7f968de12f40 JobId=0 JobStatus=R jcr=0x55eae96c6da8 name=*JobMonitor*.2025-12-18_22.57.48_01
use_count=1 killable=0
JobType=I JobLevel=
sched_time=18-Dec-2025 22:57 start_time=18-Dec-2025 22:57
end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x55eae95f64e8 rstore=(nil) wjcr=(nil) client=0x55eae95f3788 reschedule_count=0 SD_msg_chan_started=0
threadid=0x7f9626ffd640 JobId=61 JobStatus=R jcr=0x7f963c015e88 name=SullustBackup.2025-12-19_00.53.46_20
use_count=2 killable=1
JobType=B JobLevel=F
sched_time=19-Dec-2025 00:53 start_time=19-Dec-2025 00:53
end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00
db=0x7f963c024e88 db_batch=(nil) batch_started=0
wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=1
BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true
cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384
RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0
threadid=0x7f9686ffd640 JobId=67 JobStatus=c jcr=0x7f967c01c908 name=SullustBackup.2025-12-19_01.00.00_17
use_count=1 killable=0
JobType=B JobLevel=F
sched_time=19-Dec-2025 01:00 start_time=19-Dec-2025 01:00
end_time=01-Jan-1970 01:00 wait_time=19-Dec-2025 01:00
db=0x7f963c024e88 db_batch=(nil) batch_started=0
wstore=0x55eae95f6ab8 rstore=(nil) wjcr=(nil) client=0x55eae95f45e8 reschedule_count=0 SD_msg_chan_started=0
BDB=0x7f963c024e88 db_name=bacula db_user=bacula connected=true
cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=6384
RWLOCK=0x7f963c024ea0 w_active=0 w_wait=0
threadid=0x7f9607fff640 JobId=0 JobStatus=R jcr=0x7f962800b328 name=-Console-.2025-12-19_20.11.56_53
use_count=1 killable=0
JobType=U JobLevel=
sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11
end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0
threadid=0x7f9665ffb640 JobId=0 JobStatus=R jcr=0x7f962c00b6f8 name=-Console-.2025-12-19_20.11.57_20
use_count=1 killable=0
JobType=U JobLevel=
sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11
end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0
threadid=0x7f9606ffd640 JobId=0 JobStatus=R jcr=0x7f963c00f5f8 name=-Console-.2025-12-19_20.11.58_37
use_count=1 killable=0
JobType=U JobLevel=
sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11
end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0
threadid=0x7f9627fff640 JobId=0 JobStatus=R jcr=0x7f963800db68 name=-Console-.2025-12-19_20.11.58_38
use_count=1 killable=0
JobType=U JobLevel=
sched_time=19-Dec-2025 20:11 start_time=19-Dec-2025 20:11
end_time=01-Jan-1970 01:00 wait_time=01-Jan-1970 01:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x7f963802e638 rstore=(nil) wjcr=(nil) client=0x7f9638024a88 reschedule_count=0 SD_msg_chan_started=0
List plugins. Hook count=0
On Torsdag, December 18, 2025 15:38 CET, Martin Simmons <ma...@li...> wrote:
> You will need to install gdb as well, so it can get backtraces.
>
> __Martin
>
>
> >>>>> On Thu, 18 Dec 2025 14:41:46 +0100, Martin Juhl Prendergast said:
> >
> > Hi Arno
> >
> > Traceback is inserted below..
> >
> > Original I installed the packages from the EPEL9 repository, but when I had the problem there, I rebuilt the 15.0.3 package from Fedora 44, on RHEL9... only to see the same issue..
> >
> > I have started the storage daemon in debug mode, and is waiting to see debug for that, the next time it crashes..
> >
> > Regards
> >
> > If you need any more info, please
> >
> > Check the log files for more information.
> >
> > Please install a debugger (gdb) to receive a traceback.
> > Attempt to dump locks
> > threadid=0x7fc5477fe640 max=1 current=-1
> > threadid=0x7fc564ff9640 max=1 current=-1
> > threadid=0x7fc5467fc640 max=1 current=-1
> > threadid=0x7fc547fff640 max=2 current=-1
> > threadid=0x7fc5457fa640 max=1 current=-1
> > threadid=0x7fc545ffb640 max=2 current=-1
> > threadid=0x7fc59d80e640 max=1 current=-1
> > threadid=0x7fc59e00f640 max=2 current=-1
> > threadid=0x7fc59e810640 max=0 current=-1
> > threadid=0x7fc59f1ff640 max=0 current=-1
> > threadid=0x7fc5a000bf40 max=1 current=-1
> > Attempt to dump current JCRs. njcrs=3
> > threadid=0x7fc5a000bf40 JobId=0 JobStatus=R jcr=0x558b1d983008 name=*JobMonitor*.2025-12-18_11.30.53_01
> > use_count=1 killable=0
> > JobType=I JobLevel=
> > sched_time=18-Dec-2025 11:30 start_time=18-Dec-2025 11:30
> > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
> > db=(nil) db_batch=(nil) batch_started=0
> > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0
> > threadid=0x7fc545ffb640 JobId=54 JobStatus=R jcr=0x7fc5700187b8 name=SullustBackup.2025-12-18_11.35.50_45
> > use_count=2 killable=1
> > JobType=B JobLevel=F
> > sched_time=18-Dec-2025 11:35 start_time=18-Dec-2025 11:35
> > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
> > db=0x7fc570027348 db_batch=(nil) batch_started=0
> > wstore=0x558b1d8afab8 rstore=(nil) wjcr=(nil) client=0x558b1d8ad5e8 reschedule_count=0 SD_msg_chan_started=1
> > BDB=0x7fc570027348 db_name=bacula db_user=bacula connected=true
> > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=16
> > RWLOCK=0x7fc570027360 w_active=0 w_wait=0
> > threadid=0x7fc547fff640 JobId=0 JobStatus=R jcr=0x7fc53400b098 name=-Console-.2025-12-18_11.36.53_00
> > use_count=1 killable=0
> > JobType=U JobLevel=
> > sched_time=18-Dec-2025 11:36 start_time=18-Dec-2025 11:36
> > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
> > db=(nil) db_batch=(nil) batch_started=0
> > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0
> > List plugins. Hook count=0
> >
> >
> > On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <bac...@li...> wrote:
> >
> > > Hi Martin,
> > >
> > > Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast:
> > > > Using Debug on bacula-dir, I get this:
> > > >
> > > > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25)
> > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan
> > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I
> > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation
> > >
> > > Definitely deserves a thorough investigation. It's unlilkely to be
> > > caused by configuration.
> > >
> > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958]
> > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/
> > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation
> > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula
> > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked...
> > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback
> > >
> > > That file will now be relevant.
> > >
> > > /var/spool/bacula/bacula.2464062.traceback should contain information
> > > the developers can work with.
> > >
> > > Also interesting will be to know where you installed the packages from,
> > > or how you built the software.
> > >
> > > Cheers,
> > >
> > > Arno
> > >
> > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'.
> > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak.
> > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1.
> > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director.
> > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak.
> > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director.
> > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected
> > > >
> > > >
> > > > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote:
> > > >
> > > >> Also.. on the server running bacula/bacularis I get:
> > > >>
> > > >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000]
> > > >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping.
> > > >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000]
> > > >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0)
> > > >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00
> > > >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000]
> > > >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0)
> > > >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b
> > > >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000]
> > > >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000]
> > > >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping.
> > > >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000]
> > > >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000]
> > > >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0)
> > > >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b
> > > >>
> > > >>
> > > >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote:
> > > >>
> > > >>> Oh, I can see that..
> > > >>>
> > > >>> The storage daemon says:
> > > >>>
> > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
> > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director.
> > > >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
> > > >>>
> > > >>> Config:
> > > >>>
> > > >>> Director {
> > > >>> Name = "bacula-dir"
> > > >>> Password = "@@SD_PASSWORD@@"
> > > >>> }
> > > >>> Director {
> > > >>> Name = "bacula-mon"
> > > >>> Password = "@@MON_SD_PASSWORD@@"
> > > >>> Monitor = yes
> > > >>> }
> > > >>> Storage {
> > > >>> Name = "bacula-sd"
> > > >>> WorkingDirectory = "/var/spool/bacula"
> > > >>> PidDirectory = "/var/run"
> > > >>> PluginDirectory = "/usr/lib64/bacula"
> > > >>> MaximumConcurrentJobs = 20
> > > >>> }
> > > >>> Device {
> > > >>> Name = "AlwaysIncrement"
> > > >>> Description = ""
> > > >>> MediaType = "AlwaysIncrement"
> > > >>> DeviceType = "File"
> > > >>> ArchiveDevice = "/home/bacula/autoincrement"
> > > >>> RemovableMedia = no
> > > >>> RandomAccess = yes
> > > >>> AutomaticMount = yes
> > > >>> LabelMedia = yes
> > > >>> Autochanger = no
> > > >>> ReadOnly = no
> > > >>> MaximumConcurrentJobs = 5
> > > >>> DriveIndex = 0
> > > >>> }
> > > >>> Device {
> > > >>> Name = "FileChgr1-Dev1"
> > > >>> MediaType = "File1"
> > > >>> ArchiveDevice = "/tmp"
> > > >>> RemovableMedia = no
> > > >>> RandomAccess = yes
> > > >>> AutomaticMount = yes
> > > >>> LabelMedia = yes
> > > >>> AlwaysOpen = no
> > > >>> MaximumConcurrentJobs = 5
> > > >>> }
> > > >>> Device {
> > > >>> Name = "FileChgr1-Dev2"
> > > >>> MediaType = "File1"
> > > >>> ArchiveDevice = "/tmp"
> > > >>> RemovableMedia = no
> > > >>> RandomAccess = yes
> > > >>> AutomaticMount = yes
> > > >>> LabelMedia = yes
> > > >>> AlwaysOpen = no
> > > >>> MaximumConcurrentJobs = 5
> > > >>> }
> > > >>> Device {
> > > >>> Name = "FileChgr2-Dev1"
> > > >>> MediaType = "File2"
> > > >>> ArchiveDevice = "/tmp"
> > > >>> RemovableMedia = no
> > > >>> RandomAccess = yes
> > > >>> AutomaticMount = yes
> > > >>> LabelMedia = yes
> > > >>> AlwaysOpen = no
> > > >>> MaximumConcurrentJobs = 5
> > > >>> }
> > > >>> Device {
> > > >>> Name = "FileChgr2-Dev2"
> > > >>> MediaType = "File2"
> > > >>> ArchiveDevice = "/tmp"
> > > >>> RemovableMedia = no
> > > >>> RandomAccess = yes
> > > >>> AutomaticMount = yes
> > > >>> LabelMedia = yes
> > > >>> AlwaysOpen = no
> > > >>> MaximumConcurrentJobs = 5
> > > >>> }
> > > >>> Messages {
> > > >>> Name = "Standard"
> > > >>> Director = bacula-dir = All
> > > >>> }
> > > >>> Autochanger {
> > > >>> Name = "FileChgr1"
> > > >>> Device = "FileChgr1-Dev1"
> > > >>> Device = "FileChgr1-Dev2"
> > > >>> ChangerDevice = "/dev/null"
> > > >>> ChangerCommand = ""
> > > >>> }
> > > >>> Autochanger {
> > > >>> Name = "FileChgr2"
> > > >>> Device = "FileChgr2-Dev1"
> > > >>> Device = "FileChgr2-Dev2"
> > > >>> ChangerDevice = "/dev/null"
> > > >>> ChangerCommand = ""
> > > >>> }
> > > >>> Device {
> > > >>> DeviceType = "File"
> > > >>> RemovableMedia = no
> > > >>> AutomaticMount = yes
> > > >>> LabelMedia = yes
> > > >>> MaximumConcurrentJobs = 5
> > > >>> RandomAccess = yes
> > > >>> Name = "Consolidate"
> > > >>> Description = ""
> > > >>> DriveIndex = 0
> > > >>> ArchiveDevice = "/home/bacula/consolidate"
> > > >>> MediaType = "Consolidate"
> > > >>> ReadOnly = no
> > > >>> Autochanger = no
> > > >>> }
> > > >>>
> > > >>> Please say if I need to provide more configuration
> > > >>>
> > > >>> /Martin
> > > >>>
> > > >>>
> > > >>> Martin,
> > > >>>
> > > >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says".
> > > >>>
> > > >>> Regards,
> > > >>> Robert Gerber
> > > >>> 402-237-8692
> > > >>> ro...@cr...
> > > >>>
> > > >>>
> > > >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote:
> > > >>>
> > > >>> Hi guys
> > > >>>
> > > >>> Hope someone can help me..
> > > >>>
> > > >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10..
> > > >>>
> > > >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup.
> > > >>>
> > > >>> On the hosts I get:
> > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted.
> > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer
> > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103
> > > >>>
> > > >>>
> > > >>>
> > > >>> The Storage daemon says
> > > >>>
> > > >>>
> > > >>> _______________________________________________
> > > >>> Bacula-users mailing list
> > > >>> Bac...@li...
> > > >>> https://lists.sourceforge.net/lists/listinfo/bacula-users
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Bacula-users mailing list
> > > > Bac...@li...
> > > > https://lists.sourceforge.net/lists/listinfo/bacula-users
> > >
> > > --
> > > Arno Lehmann
> > >
> > > IT-Service Lehmann
> > > Sandstr. 6, 49080 Osnabrück
> > >
> > >
> > >
> > > _______________________________________________
> > > Bacula-users mailing list
> > > Bac...@li...
> > > https://lists.sourceforge.net/lists/listinfo/bacula-users
> >
> >
> >
> > _______________________________________________
> > Bacula-users mailing list
> > Bac...@li...
> > https://lists.sourceforge.net/lists/listinfo/bacula-users
> >
|
|
From: Martin S. <ma...@li...> - 2025-12-18 14:39:03
|
You will need to install gdb as well, so it can get backtraces. __Martin >>>>> On Thu, 18 Dec 2025 14:41:46 +0100, Martin Juhl Prendergast said: > > Hi Arno > > Traceback is inserted below.. > > Original I installed the packages from the EPEL9 repository, but when I had the problem there, I rebuilt the 15.0.3 package from Fedora 44, on RHEL9... only to see the same issue.. > > I have started the storage daemon in debug mode, and is waiting to see debug for that, the next time it crashes.. > > Regards > > If you need any more info, please > > Check the log files for more information. > > Please install a debugger (gdb) to receive a traceback. > Attempt to dump locks > threadid=0x7fc5477fe640 max=1 current=-1 > threadid=0x7fc564ff9640 max=1 current=-1 > threadid=0x7fc5467fc640 max=1 current=-1 > threadid=0x7fc547fff640 max=2 current=-1 > threadid=0x7fc5457fa640 max=1 current=-1 > threadid=0x7fc545ffb640 max=2 current=-1 > threadid=0x7fc59d80e640 max=1 current=-1 > threadid=0x7fc59e00f640 max=2 current=-1 > threadid=0x7fc59e810640 max=0 current=-1 > threadid=0x7fc59f1ff640 max=0 current=-1 > threadid=0x7fc5a000bf40 max=1 current=-1 > Attempt to dump current JCRs. njcrs=3 > threadid=0x7fc5a000bf40 JobId=0 JobStatus=R jcr=0x558b1d983008 name=*JobMonitor*.2025-12-18_11.30.53_01 > use_count=1 killable=0 > JobType=I JobLevel= > sched_time=18-Dec-2025 11:30 start_time=18-Dec-2025 11:30 > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > threadid=0x7fc545ffb640 JobId=54 JobStatus=R jcr=0x7fc5700187b8 name=SullustBackup.2025-12-18_11.35.50_45 > use_count=2 killable=1 > JobType=B JobLevel=F > sched_time=18-Dec-2025 11:35 start_time=18-Dec-2025 11:35 > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > db=0x7fc570027348 db_batch=(nil) batch_started=0 > wstore=0x558b1d8afab8 rstore=(nil) wjcr=(nil) client=0x558b1d8ad5e8 reschedule_count=0 SD_msg_chan_started=1 > BDB=0x7fc570027348 db_name=bacula db_user=bacula connected=true > cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=16 > RWLOCK=0x7fc570027360 w_active=0 w_wait=0 > threadid=0x7fc547fff640 JobId=0 JobStatus=R jcr=0x7fc53400b098 name=-Console-.2025-12-18_11.36.53_00 > use_count=1 killable=0 > JobType=U JobLevel= > sched_time=18-Dec-2025 11:36 start_time=18-Dec-2025 11:36 > end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0 > List plugins. Hook count=0 > > > On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <bac...@li...> wrote: > > > Hi Martin, > > > > Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast: > > > Using Debug on bacula-dir, I get this: > > > > > > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25) > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan > > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > > Definitely deserves a thorough investigation. It's unlilkely to be > > caused by configuration. > > > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958] > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/ > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked... > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback > > > > That file will now be relevant. > > > > /var/spool/bacula/bacula.2464062.traceback should contain information > > the developers can work with. > > > > Also interesting will be to know where you installed the packages from, > > or how you built the software. > > > > Cheers, > > > > Arno > > > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'. > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1. > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director. > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director. > > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected > > > > > > > > > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > > > >> Also.. on the server running bacula/bacularis I get: > > >> > > >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000] > > >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping. > > >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000] > > >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0) > > >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00 > > >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000] > > >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0) > > >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000] > > >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000] > > >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping. > > >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000] > > >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000] > > >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0) > > >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > >> > > >> > > >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > >> > > >>> Oh, I can see that.. > > >>> > > >>> The storage daemon says: > > >>> > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. > > >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > >>> > > >>> Config: > > >>> > > >>> Director { > > >>> Name = "bacula-dir" > > >>> Password = "@@SD_PASSWORD@@" > > >>> } > > >>> Director { > > >>> Name = "bacula-mon" > > >>> Password = "@@MON_SD_PASSWORD@@" > > >>> Monitor = yes > > >>> } > > >>> Storage { > > >>> Name = "bacula-sd" > > >>> WorkingDirectory = "/var/spool/bacula" > > >>> PidDirectory = "/var/run" > > >>> PluginDirectory = "/usr/lib64/bacula" > > >>> MaximumConcurrentJobs = 20 > > >>> } > > >>> Device { > > >>> Name = "AlwaysIncrement" > > >>> Description = "" > > >>> MediaType = "AlwaysIncrement" > > >>> DeviceType = "File" > > >>> ArchiveDevice = "/home/bacula/autoincrement" > > >>> RemovableMedia = no > > >>> RandomAccess = yes > > >>> AutomaticMount = yes > > >>> LabelMedia = yes > > >>> Autochanger = no > > >>> ReadOnly = no > > >>> MaximumConcurrentJobs = 5 > > >>> DriveIndex = 0 > > >>> } > > >>> Device { > > >>> Name = "FileChgr1-Dev1" > > >>> MediaType = "File1" > > >>> ArchiveDevice = "/tmp" > > >>> RemovableMedia = no > > >>> RandomAccess = yes > > >>> AutomaticMount = yes > > >>> LabelMedia = yes > > >>> AlwaysOpen = no > > >>> MaximumConcurrentJobs = 5 > > >>> } > > >>> Device { > > >>> Name = "FileChgr1-Dev2" > > >>> MediaType = "File1" > > >>> ArchiveDevice = "/tmp" > > >>> RemovableMedia = no > > >>> RandomAccess = yes > > >>> AutomaticMount = yes > > >>> LabelMedia = yes > > >>> AlwaysOpen = no > > >>> MaximumConcurrentJobs = 5 > > >>> } > > >>> Device { > > >>> Name = "FileChgr2-Dev1" > > >>> MediaType = "File2" > > >>> ArchiveDevice = "/tmp" > > >>> RemovableMedia = no > > >>> RandomAccess = yes > > >>> AutomaticMount = yes > > >>> LabelMedia = yes > > >>> AlwaysOpen = no > > >>> MaximumConcurrentJobs = 5 > > >>> } > > >>> Device { > > >>> Name = "FileChgr2-Dev2" > > >>> MediaType = "File2" > > >>> ArchiveDevice = "/tmp" > > >>> RemovableMedia = no > > >>> RandomAccess = yes > > >>> AutomaticMount = yes > > >>> LabelMedia = yes > > >>> AlwaysOpen = no > > >>> MaximumConcurrentJobs = 5 > > >>> } > > >>> Messages { > > >>> Name = "Standard" > > >>> Director = bacula-dir = All > > >>> } > > >>> Autochanger { > > >>> Name = "FileChgr1" > > >>> Device = "FileChgr1-Dev1" > > >>> Device = "FileChgr1-Dev2" > > >>> ChangerDevice = "/dev/null" > > >>> ChangerCommand = "" > > >>> } > > >>> Autochanger { > > >>> Name = "FileChgr2" > > >>> Device = "FileChgr2-Dev1" > > >>> Device = "FileChgr2-Dev2" > > >>> ChangerDevice = "/dev/null" > > >>> ChangerCommand = "" > > >>> } > > >>> Device { > > >>> DeviceType = "File" > > >>> RemovableMedia = no > > >>> AutomaticMount = yes > > >>> LabelMedia = yes > > >>> MaximumConcurrentJobs = 5 > > >>> RandomAccess = yes > > >>> Name = "Consolidate" > > >>> Description = "" > > >>> DriveIndex = 0 > > >>> ArchiveDevice = "/home/bacula/consolidate" > > >>> MediaType = "Consolidate" > > >>> ReadOnly = no > > >>> Autochanger = no > > >>> } > > >>> > > >>> Please say if I need to provide more configuration > > >>> > > >>> /Martin > > >>> > > >>> > > >>> Martin, > > >>> > > >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says". > > >>> > > >>> Regards, > > >>> Robert Gerber > > >>> 402-237-8692 > > >>> ro...@cr... > > >>> > > >>> > > >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: > > >>> > > >>> Hi guys > > >>> > > >>> Hope someone can help me.. > > >>> > > >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. > > >>> > > >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. > > >>> > > >>> On the hosts I get: > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer > > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 > > >>> > > >>> > > >>> > > >>> The Storage daemon says > > >>> > > >>> > > >>> _______________________________________________ > > >>> Bacula-users mailing list > > >>> Bac...@li... > > >>> https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > > > > > _______________________________________________ > > > Bacula-users mailing list > > > Bac...@li... > > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > -- > > Arno Lehmann > > > > IT-Service Lehmann > > Sandstr. 6, 49080 Osnabrück > > > > > > > > _______________________________________________ > > Bacula-users mailing list > > Bac...@li... > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users > |
|
From: Martin J. P. <m...@rt...> - 2025-12-18 14:28:09
|
Hi again Another update, when the problem happens, even with debugging enabled, this is all the SD gives: Dec 18 14:24:43 degobah.mrmeee.dk bacula-sd[2478367]: bacula-sd: message.c:1841-55 bsock.c:275 Socket has errors=1 on call to client:37.187.95.186:9103 Dec 18 14:24:43 degobah.mrmeee.dk bacula-sd[2478367]: bacula-sd: record_write.c:236-55 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. Dec 18 14:24:43 degobah.mrmeee.dk bacula-sd[2478367]: bacula-sd: events.c:48-55 Events: code=SJ0002 daemon=bacula-sd ref=0x7fceb800b098 type=job source=bacula-dir text=Job End jobid=55 job=SullustBackup.2025-12-18_11.53.44_47 status=f Dec 18 14:24:43 degobah.mrmeee.dk bacula-sd[2478367]: bacula-sd: message.c:1841-55 bsock.c:275 Socket has errors=1 on call to client:37.187.95.186:9103 Regards On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <bac...@li...> wrote: > Hi Martin, > > Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast: > > Using Debug on bacula-dir, I get this: > > > > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25) > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan > > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > Definitely deserves a thorough investigation. It's unlilkely to be > caused by configuration. > > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958] > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/ > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked... > > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback > > That file will now be relevant. > > /var/spool/bacula/bacula.2464062.traceback should contain information > the developers can work with. > > Also interesting will be to know where you installed the packages from, > or how you built the software. > > Cheers, > > Arno > > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'. > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1. > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director. > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director. > > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected > > > > > > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > >> Also.. on the server running bacula/bacularis I get: > >> > >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000] > >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping. > >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000] > >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0) > >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00 > >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000] > >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0) > >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000] > >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000] > >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping. > >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000] > >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000] > >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0) > >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > >> > >> > >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > >> > >>> Oh, I can see that.. > >>> > >>> The storage daemon says: > >>> > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. > >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > >>> > >>> Config: > >>> > >>> Director { > >>> Name = "bacula-dir" > >>> Password = "@@SD_PASSWORD@@" > >>> } > >>> Director { > >>> Name = "bacula-mon" > >>> Password = "@@MON_SD_PASSWORD@@" > >>> Monitor = yes > >>> } > >>> Storage { > >>> Name = "bacula-sd" > >>> WorkingDirectory = "/var/spool/bacula" > >>> PidDirectory = "/var/run" > >>> PluginDirectory = "/usr/lib64/bacula" > >>> MaximumConcurrentJobs = 20 > >>> } > >>> Device { > >>> Name = "AlwaysIncrement" > >>> Description = "" > >>> MediaType = "AlwaysIncrement" > >>> DeviceType = "File" > >>> ArchiveDevice = "/home/bacula/autoincrement" > >>> RemovableMedia = no > >>> RandomAccess = yes > >>> AutomaticMount = yes > >>> LabelMedia = yes > >>> Autochanger = no > >>> ReadOnly = no > >>> MaximumConcurrentJobs = 5 > >>> DriveIndex = 0 > >>> } > >>> Device { > >>> Name = "FileChgr1-Dev1" > >>> MediaType = "File1" > >>> ArchiveDevice = "/tmp" > >>> RemovableMedia = no > >>> RandomAccess = yes > >>> AutomaticMount = yes > >>> LabelMedia = yes > >>> AlwaysOpen = no > >>> MaximumConcurrentJobs = 5 > >>> } > >>> Device { > >>> Name = "FileChgr1-Dev2" > >>> MediaType = "File1" > >>> ArchiveDevice = "/tmp" > >>> RemovableMedia = no > >>> RandomAccess = yes > >>> AutomaticMount = yes > >>> LabelMedia = yes > >>> AlwaysOpen = no > >>> MaximumConcurrentJobs = 5 > >>> } > >>> Device { > >>> Name = "FileChgr2-Dev1" > >>> MediaType = "File2" > >>> ArchiveDevice = "/tmp" > >>> RemovableMedia = no > >>> RandomAccess = yes > >>> AutomaticMount = yes > >>> LabelMedia = yes > >>> AlwaysOpen = no > >>> MaximumConcurrentJobs = 5 > >>> } > >>> Device { > >>> Name = "FileChgr2-Dev2" > >>> MediaType = "File2" > >>> ArchiveDevice = "/tmp" > >>> RemovableMedia = no > >>> RandomAccess = yes > >>> AutomaticMount = yes > >>> LabelMedia = yes > >>> AlwaysOpen = no > >>> MaximumConcurrentJobs = 5 > >>> } > >>> Messages { > >>> Name = "Standard" > >>> Director = bacula-dir = All > >>> } > >>> Autochanger { > >>> Name = "FileChgr1" > >>> Device = "FileChgr1-Dev1" > >>> Device = "FileChgr1-Dev2" > >>> ChangerDevice = "/dev/null" > >>> ChangerCommand = "" > >>> } > >>> Autochanger { > >>> Name = "FileChgr2" > >>> Device = "FileChgr2-Dev1" > >>> Device = "FileChgr2-Dev2" > >>> ChangerDevice = "/dev/null" > >>> ChangerCommand = "" > >>> } > >>> Device { > >>> DeviceType = "File" > >>> RemovableMedia = no > >>> AutomaticMount = yes > >>> LabelMedia = yes > >>> MaximumConcurrentJobs = 5 > >>> RandomAccess = yes > >>> Name = "Consolidate" > >>> Description = "" > >>> DriveIndex = 0 > >>> ArchiveDevice = "/home/bacula/consolidate" > >>> MediaType = "Consolidate" > >>> ReadOnly = no > >>> Autochanger = no > >>> } > >>> > >>> Please say if I need to provide more configuration > >>> > >>> /Martin > >>> > >>> > >>> Martin, > >>> > >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says". > >>> > >>> Regards, > >>> Robert Gerber > >>> 402-237-8692 > >>> ro...@cr... > >>> > >>> > >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: > >>> > >>> Hi guys > >>> > >>> Hope someone can help me.. > >>> > >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. > >>> > >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. > >>> > >>> On the hosts I get: > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer > >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 > >>> > >>> > >>> > >>> The Storage daemon says > >>> > >>> > >>> _______________________________________________ > >>> Bacula-users mailing list > >>> Bac...@li... > >>> https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > > > > > _______________________________________________ > > Bacula-users mailing list > > Bac...@li... > > https://lists.sourceforge.net/lists/listinfo/bacula-users > > -- > Arno Lehmann > > IT-Service Lehmann > Sandstr. 6, 49080 Osnabrück > > > > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users |
|
From: Martin J. P. <m...@rt...> - 2025-12-18 13:42:05
|
Hi Arno
Traceback is inserted below..
Original I installed the packages from the EPEL9 repository, but when I had the problem there, I rebuilt the 15.0.3 package from Fedora 44, on RHEL9... only to see the same issue..
I have started the storage daemon in debug mode, and is waiting to see debug for that, the next time it crashes..
Regards
If you need any more info, please
Check the log files for more information.
Please install a debugger (gdb) to receive a traceback.
Attempt to dump locks
threadid=0x7fc5477fe640 max=1 current=-1
threadid=0x7fc564ff9640 max=1 current=-1
threadid=0x7fc5467fc640 max=1 current=-1
threadid=0x7fc547fff640 max=2 current=-1
threadid=0x7fc5457fa640 max=1 current=-1
threadid=0x7fc545ffb640 max=2 current=-1
threadid=0x7fc59d80e640 max=1 current=-1
threadid=0x7fc59e00f640 max=2 current=-1
threadid=0x7fc59e810640 max=0 current=-1
threadid=0x7fc59f1ff640 max=0 current=-1
threadid=0x7fc5a000bf40 max=1 current=-1
Attempt to dump current JCRs. njcrs=3
threadid=0x7fc5a000bf40 JobId=0 JobStatus=R jcr=0x558b1d983008 name=*JobMonitor*.2025-12-18_11.30.53_01
use_count=1 killable=0
JobType=I JobLevel=
sched_time=18-Dec-2025 11:30 start_time=18-Dec-2025 11:30
end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0
threadid=0x7fc545ffb640 JobId=54 JobStatus=R jcr=0x7fc5700187b8 name=SullustBackup.2025-12-18_11.35.50_45
use_count=2 killable=1
JobType=B JobLevel=F
sched_time=18-Dec-2025 11:35 start_time=18-Dec-2025 11:35
end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
db=0x7fc570027348 db_batch=(nil) batch_started=0
wstore=0x558b1d8afab8 rstore=(nil) wjcr=(nil) client=0x558b1d8ad5e8 reschedule_count=0 SD_msg_chan_started=1
BDB=0x7fc570027348 db_name=bacula db_user=bacula connected=true
cmd="UPDATE Client SET AutoPrune=1,FileRetention=5184000,JobRetention=15552000,Uname='15.0.3 (25Mar25) x86_64-redhat-linux-gnu,redhat,Enterprise 9.6',Plugins='bpipe(2),cdp(0.1),docker(1.2.1),antivirus(1)' WHERE Name='sullust.outerrim.lan'" changes=16
RWLOCK=0x7fc570027360 w_active=0 w_wait=0
threadid=0x7fc547fff640 JobId=0 JobStatus=R jcr=0x7fc53400b098 name=-Console-.2025-12-18_11.36.53_00
use_count=1 killable=0
JobType=U JobLevel=
sched_time=18-Dec-2025 11:36 start_time=18-Dec-2025 11:36
end_time=01-Jan-1970 00:00 wait_time=01-Jan-1970 00:00
db=(nil) db_batch=(nil) batch_started=0
wstore=0x558b1d8af4e8 rstore=(nil) wjcr=(nil) client=0x558b1d8ac788 reschedule_count=0 SD_msg_chan_started=0
List plugins. Hook count=0
On Torsdag, December 18, 2025 13:04 CET, Arno Lehmann via Bacula-users <bac...@li...> wrote:
> Hi Martin,
>
> Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast:
> > Using Debug on bacula-dir, I get this:
> >
> > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25)
> > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan
> > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation
>
> Definitely deserves a thorough investigation. It's unlilkely to be
> caused by configuration.
>
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958]
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked...
> > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback
>
> That file will now be relevant.
>
> /var/spool/bacula/bacula.2464062.traceback should contain information
> the developers can work with.
>
> Also interesting will be to know where you installed the packages from,
> or how you built the software.
>
> Cheers,
>
> Arno
>
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=|
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=|
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak.
> > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director.
> > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected
> >
> >
> > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote:
> >
> >> Also.. on the server running bacula/bacularis I get:
> >>
> >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000]
> >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping.
> >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000]
> >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0)
> >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00
> >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000]
> >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0)
> >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b
> >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000]
> >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000]
> >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping.
> >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000]
> >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000]
> >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0)
> >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b
> >>
> >>
> >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote:
> >>
> >>> Oh, I can see that..
> >>>
> >>> The storage daemon says:
> >>>
> >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
> >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director.
> >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
> >>>
> >>> Config:
> >>>
> >>> Director {
> >>> Name = "bacula-dir"
> >>> Password = "@@SD_PASSWORD@@"
> >>> }
> >>> Director {
> >>> Name = "bacula-mon"
> >>> Password = "@@MON_SD_PASSWORD@@"
> >>> Monitor = yes
> >>> }
> >>> Storage {
> >>> Name = "bacula-sd"
> >>> WorkingDirectory = "/var/spool/bacula"
> >>> PidDirectory = "/var/run"
> >>> PluginDirectory = "/usr/lib64/bacula"
> >>> MaximumConcurrentJobs = 20
> >>> }
> >>> Device {
> >>> Name = "AlwaysIncrement"
> >>> Description = ""
> >>> MediaType = "AlwaysIncrement"
> >>> DeviceType = "File"
> >>> ArchiveDevice = "/home/bacula/autoincrement"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> Autochanger = no
> >>> ReadOnly = no
> >>> MaximumConcurrentJobs = 5
> >>> DriveIndex = 0
> >>> }
> >>> Device {
> >>> Name = "FileChgr1-Dev1"
> >>> MediaType = "File1"
> >>> ArchiveDevice = "/tmp"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> AlwaysOpen = no
> >>> MaximumConcurrentJobs = 5
> >>> }
> >>> Device {
> >>> Name = "FileChgr1-Dev2"
> >>> MediaType = "File1"
> >>> ArchiveDevice = "/tmp"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> AlwaysOpen = no
> >>> MaximumConcurrentJobs = 5
> >>> }
> >>> Device {
> >>> Name = "FileChgr2-Dev1"
> >>> MediaType = "File2"
> >>> ArchiveDevice = "/tmp"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> AlwaysOpen = no
> >>> MaximumConcurrentJobs = 5
> >>> }
> >>> Device {
> >>> Name = "FileChgr2-Dev2"
> >>> MediaType = "File2"
> >>> ArchiveDevice = "/tmp"
> >>> RemovableMedia = no
> >>> RandomAccess = yes
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> AlwaysOpen = no
> >>> MaximumConcurrentJobs = 5
> >>> }
> >>> Messages {
> >>> Name = "Standard"
> >>> Director = bacula-dir = All
> >>> }
> >>> Autochanger {
> >>> Name = "FileChgr1"
> >>> Device = "FileChgr1-Dev1"
> >>> Device = "FileChgr1-Dev2"
> >>> ChangerDevice = "/dev/null"
> >>> ChangerCommand = ""
> >>> }
> >>> Autochanger {
> >>> Name = "FileChgr2"
> >>> Device = "FileChgr2-Dev1"
> >>> Device = "FileChgr2-Dev2"
> >>> ChangerDevice = "/dev/null"
> >>> ChangerCommand = ""
> >>> }
> >>> Device {
> >>> DeviceType = "File"
> >>> RemovableMedia = no
> >>> AutomaticMount = yes
> >>> LabelMedia = yes
> >>> MaximumConcurrentJobs = 5
> >>> RandomAccess = yes
> >>> Name = "Consolidate"
> >>> Description = ""
> >>> DriveIndex = 0
> >>> ArchiveDevice = "/home/bacula/consolidate"
> >>> MediaType = "Consolidate"
> >>> ReadOnly = no
> >>> Autochanger = no
> >>> }
> >>>
> >>> Please say if I need to provide more configuration
> >>>
> >>> /Martin
> >>>
> >>>
> >>> Martin,
> >>>
> >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says".
> >>>
> >>> Regards,
> >>> Robert Gerber
> >>> 402-237-8692
> >>> ro...@cr...
> >>>
> >>>
> >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote:
> >>>
> >>> Hi guys
> >>>
> >>> Hope someone can help me..
> >>>
> >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10..
> >>>
> >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup.
> >>>
> >>> On the hosts I get:
> >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted.
> >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer
> >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103
> >>>
> >>>
> >>>
> >>> The Storage daemon says
> >>>
> >>>
> >>> _______________________________________________
> >>> Bacula-users mailing list
> >>> Bac...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/bacula-users
> >
> >
> >
> > _______________________________________________
> > Bacula-users mailing list
> > Bac...@li...
> > https://lists.sourceforge.net/lists/listinfo/bacula-users
>
> --
> Arno Lehmann
>
> IT-Service Lehmann
> Sandstr. 6, 49080 Osnabrück
>
>
>
> _______________________________________________
> Bacula-users mailing list
> Bac...@li...
> https://lists.sourceforge.net/lists/listinfo/bacula-users
|
|
From: Arno L. <al...@it...> - 2025-12-18 12:08:20
|
Hi Martin, Am 18.12.2025 um 12:51 schrieb Martin Juhl Prendergast: > Using Debug on bacula-dir, I get this: > > Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25) > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan > Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation Definitely deserves a thorough investigation. It's unlilkely to be caused by configuration. > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958] > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/ > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked... > Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback That file will now be relevant. /var/spool/bacula/bacula.2464062.traceback should contain information the developers can work with. Also interesting will be to know where you installed the packages from, or how you built the software. Cheers, Arno > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'. > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1. > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director. > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. > Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director. > Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected > > > On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > >> Also.. on the server running bacula/bacularis I get: >> >> [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000] >> [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping. >> [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000] >> [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0) >> [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00 >> [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000] >> [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0) >> [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b >> [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000] >> [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000] >> [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping. >> [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000] >> [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000] >> [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0) >> [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b >> >> >> On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: >> >>> Oh, I can see that.. >>> >>> The storage daemon says: >>> >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 >>> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. >>> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 >>> >>> Config: >>> >>> Director { >>> Name = "bacula-dir" >>> Password = "@@SD_PASSWORD@@" >>> } >>> Director { >>> Name = "bacula-mon" >>> Password = "@@MON_SD_PASSWORD@@" >>> Monitor = yes >>> } >>> Storage { >>> Name = "bacula-sd" >>> WorkingDirectory = "/var/spool/bacula" >>> PidDirectory = "/var/run" >>> PluginDirectory = "/usr/lib64/bacula" >>> MaximumConcurrentJobs = 20 >>> } >>> Device { >>> Name = "AlwaysIncrement" >>> Description = "" >>> MediaType = "AlwaysIncrement" >>> DeviceType = "File" >>> ArchiveDevice = "/home/bacula/autoincrement" >>> RemovableMedia = no >>> RandomAccess = yes >>> AutomaticMount = yes >>> LabelMedia = yes >>> Autochanger = no >>> ReadOnly = no >>> MaximumConcurrentJobs = 5 >>> DriveIndex = 0 >>> } >>> Device { >>> Name = "FileChgr1-Dev1" >>> MediaType = "File1" >>> ArchiveDevice = "/tmp" >>> RemovableMedia = no >>> RandomAccess = yes >>> AutomaticMount = yes >>> LabelMedia = yes >>> AlwaysOpen = no >>> MaximumConcurrentJobs = 5 >>> } >>> Device { >>> Name = "FileChgr1-Dev2" >>> MediaType = "File1" >>> ArchiveDevice = "/tmp" >>> RemovableMedia = no >>> RandomAccess = yes >>> AutomaticMount = yes >>> LabelMedia = yes >>> AlwaysOpen = no >>> MaximumConcurrentJobs = 5 >>> } >>> Device { >>> Name = "FileChgr2-Dev1" >>> MediaType = "File2" >>> ArchiveDevice = "/tmp" >>> RemovableMedia = no >>> RandomAccess = yes >>> AutomaticMount = yes >>> LabelMedia = yes >>> AlwaysOpen = no >>> MaximumConcurrentJobs = 5 >>> } >>> Device { >>> Name = "FileChgr2-Dev2" >>> MediaType = "File2" >>> ArchiveDevice = "/tmp" >>> RemovableMedia = no >>> RandomAccess = yes >>> AutomaticMount = yes >>> LabelMedia = yes >>> AlwaysOpen = no >>> MaximumConcurrentJobs = 5 >>> } >>> Messages { >>> Name = "Standard" >>> Director = bacula-dir = All >>> } >>> Autochanger { >>> Name = "FileChgr1" >>> Device = "FileChgr1-Dev1" >>> Device = "FileChgr1-Dev2" >>> ChangerDevice = "/dev/null" >>> ChangerCommand = "" >>> } >>> Autochanger { >>> Name = "FileChgr2" >>> Device = "FileChgr2-Dev1" >>> Device = "FileChgr2-Dev2" >>> ChangerDevice = "/dev/null" >>> ChangerCommand = "" >>> } >>> Device { >>> DeviceType = "File" >>> RemovableMedia = no >>> AutomaticMount = yes >>> LabelMedia = yes >>> MaximumConcurrentJobs = 5 >>> RandomAccess = yes >>> Name = "Consolidate" >>> Description = "" >>> DriveIndex = 0 >>> ArchiveDevice = "/home/bacula/consolidate" >>> MediaType = "Consolidate" >>> ReadOnly = no >>> Autochanger = no >>> } >>> >>> Please say if I need to provide more configuration >>> >>> /Martin >>> >>> >>> Martin, >>> >>> It looks like your message was cut off. It doesn't have any information after "The storage daemon says". >>> >>> Regards, >>> Robert Gerber >>> 402-237-8692 >>> ro...@cr... >>> >>> >>> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: >>> >>> Hi guys >>> >>> Hope someone can help me.. >>> >>> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. >>> >>> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. >>> >>> On the hosts I get: >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer >>> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 >>> >>> >>> >>> The Storage daemon says >>> >>> >>> _______________________________________________ >>> Bacula-users mailing list >>> Bac...@li... >>> https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück |
|
From: Martin J. P. <m...@rt...> - 2025-12-18 11:52:18
|
Using Debug on bacula-dir, I get this: Dec 18 11:30:53 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DD0001 daemon=bacula-dir ref=0x238d type=daemon source=*Director* text=Director startup 15.0.3 (25Mar25) Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0004 daemon=bacula-dir ref=0x7fc57000edb8 type=command source=*Console* text=run job=SullustBackup fileset=SullustFileset client=sullust.outerrim.lan Dec 18 11:35:50 degobah.mrmeee.dk bacula-dir[2464062]: bacula-dir: events.c:48-0 Events: code=DJ0001 daemon=bacula-dir ref=0x7fc5700187b8 type=job source=*Director* text=Job Creation jobid=54 name=SullustBackup.2025-12-18_11.35.50_45 type=B level=I Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! bacula-dir, bacula-dir got signal 11 - Segmentation violation at 18-Dec-2025 11:36:53. Attempting traceback. thread#=[2958] Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Kaboom! exepath=/usr/sbin/ Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: Bacula interrupted by signal 11: Segmentation violation Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2468313]: Calling: /usr/sbin/btraceback /usr/sbin/bacula-dir 2464062 /var/spool/bacula Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: It looks like the traceback worked... Dec 18 11:36:53 degobah.mrmeee.dk bacula-dir[2464062]: LockDump: /var/spool/bacula/bacula.2464062.traceback Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Main process exited, code=dumped, status=11/SEGV Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/make_catalog_backup.pl MyCatalog type=| Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: bacula-dir: dird_conf.c:2594-0 runscript cmd=/usr/libexec/bacula/delete_catalog_backup type=| Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Failed with result 'core-dump'. Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Scheduled restart job, restart counter is at 1. Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Stopped Bacula Director. Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: bacula-dir.service: Consumed 5.532s CPU time, 9.5M memory peak. Dec 18 11:37:53 degobah.mrmeee.dk systemd[1]: Started Bacula Director. Dec 18 11:37:54 degobah.mrmeee.dk bacula-dir[2469593]: Postgresql and system timezone mismatch detected On Torsdag, December 18, 2025 12:27 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > Also.. on the server running bacula/bacularis I get: > > [989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000] > [991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping. > [998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000] > [1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0) > [1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00 > [1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000] > [1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0) > [1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > [1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000] > [1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000] > [1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping. > [1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000] > [1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000] > [1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0) > [1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b > > > On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote: > > > Oh, I can see that.. > > > > The storage daemon says: > > > > Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director. > > Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103 > > > > Config: > > > > Director { > > Name = "bacula-dir" > > Password = "@@SD_PASSWORD@@" > > } > > Director { > > Name = "bacula-mon" > > Password = "@@MON_SD_PASSWORD@@" > > Monitor = yes > > } > > Storage { > > Name = "bacula-sd" > > WorkingDirectory = "/var/spool/bacula" > > PidDirectory = "/var/run" > > PluginDirectory = "/usr/lib64/bacula" > > MaximumConcurrentJobs = 20 > > } > > Device { > > Name = "AlwaysIncrement" > > Description = "" > > MediaType = "AlwaysIncrement" > > DeviceType = "File" > > ArchiveDevice = "/home/bacula/autoincrement" > > RemovableMedia = no > > RandomAccess = yes > > AutomaticMount = yes > > LabelMedia = yes > > Autochanger = no > > ReadOnly = no > > MaximumConcurrentJobs = 5 > > DriveIndex = 0 > > } > > Device { > > Name = "FileChgr1-Dev1" > > MediaType = "File1" > > ArchiveDevice = "/tmp" > > RemovableMedia = no > > RandomAccess = yes > > AutomaticMount = yes > > LabelMedia = yes > > AlwaysOpen = no > > MaximumConcurrentJobs = 5 > > } > > Device { > > Name = "FileChgr1-Dev2" > > MediaType = "File1" > > ArchiveDevice = "/tmp" > > RemovableMedia = no > > RandomAccess = yes > > AutomaticMount = yes > > LabelMedia = yes > > AlwaysOpen = no > > MaximumConcurrentJobs = 5 > > } > > Device { > > Name = "FileChgr2-Dev1" > > MediaType = "File2" > > ArchiveDevice = "/tmp" > > RemovableMedia = no > > RandomAccess = yes > > AutomaticMount = yes > > LabelMedia = yes > > AlwaysOpen = no > > MaximumConcurrentJobs = 5 > > } > > Device { > > Name = "FileChgr2-Dev2" > > MediaType = "File2" > > ArchiveDevice = "/tmp" > > RemovableMedia = no > > RandomAccess = yes > > AutomaticMount = yes > > LabelMedia = yes > > AlwaysOpen = no > > MaximumConcurrentJobs = 5 > > } > > Messages { > > Name = "Standard" > > Director = bacula-dir = All > > } > > Autochanger { > > Name = "FileChgr1" > > Device = "FileChgr1-Dev1" > > Device = "FileChgr1-Dev2" > > ChangerDevice = "/dev/null" > > ChangerCommand = "" > > } > > Autochanger { > > Name = "FileChgr2" > > Device = "FileChgr2-Dev1" > > Device = "FileChgr2-Dev2" > > ChangerDevice = "/dev/null" > > ChangerCommand = "" > > } > > Device { > > DeviceType = "File" > > RemovableMedia = no > > AutomaticMount = yes > > LabelMedia = yes > > MaximumConcurrentJobs = 5 > > RandomAccess = yes > > Name = "Consolidate" > > Description = "" > > DriveIndex = 0 > > ArchiveDevice = "/home/bacula/consolidate" > > MediaType = "Consolidate" > > ReadOnly = no > > Autochanger = no > > } > > > > Please say if I need to provide more configuration > > > > /Martin > > > > > > Martin, > > > > It looks like your message was cut off. It doesn't have any information after "The storage daemon says". > > > > Regards, > > Robert Gerber > > 402-237-8692 > > ro...@cr... > > > > > > On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: > > > > Hi guys > > > > Hope someone can help me.. > > > > I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. > > > > I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. > > > > On the hosts I get: > > Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. > > Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer > > Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 > > > > > > > > The Storage daemon says > > > > > > _______________________________________________ > > Bacula-users mailing list > > Bac...@li... > > https://lists.sourceforge.net/lists/listinfo/bacula-users |
|
From: Martin J. P. <m...@rt...> - 2025-12-18 11:27:46
|
Also.. on the server running bacula/bacularis I get:
[989239.395576] traps: bacula-dir[716830] general protection fault ip:7f4aef66fc98 sp:7f4aec864bc8 error:0 in libbac-11.0.1.so[7f4aef649000+55000]
[991483.696278] systemd-rc-local-generator[822569]: /etc/rc.d/rc.local is not marked executable, skipping.
[998682.714339] traps: bacula-dir[825738] general protection fault ip:7effee4adee8 sp:7effe6ffcbc8 error:0 in libbac-15.0.3.so[7effee486000+66000]
[1004933.001982] bacula-dir[958919]: segfault at 10 ip 000055dcaec86a9c sp 00007fc8f6ffbd90 error 4 in bacula-dir[55dcaec7e000+8f000] likely on CPU 3 (core 3, socket 0)
[1004933.017876] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f8 48 83 ec 08 48 89 f7 48 8b 90 d0 04 00 00 0f b6 b0 6d 04 00 00 <48> 8b 4a 10 48 8b 52 70 56 8b b0 50 13 00 00 56 4c 8b 88 e8 04 00
[1028754.467741] traps: bacula-dir[958999] general protection fault ip:7f52a4ffdee8 sp:7f52a2264bc8 error:0 in libbac-15.0.3.so[7f52a4fd6000+66000]
[1031101.885483] bacula-dir[1223299]: segfault at 561038000000 ip 00007f262de9b40b sp 00007f25e7ffd070 error 4 in libc.so.6[7f262de29000+175000] likely on CPU 7 (core 3, socket 0)
[1031101.901480] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b
[1031101.909765] traps: bacula-dir[1223320] general protection fault ip:5610375d3a64 sp:7f25e57f9980 error:0 in bacula-dir[5610375c5000+8f000]
[1033627.815454] traps: bacula-dir[1223365] general protection fault ip:7f38b4094ee8 sp:7f38b1264bc8 error:0 in libbac-15.0.3.so[7f38b406d000+66000]
[1038433.757733] systemd-rc-local-generator[1305859]: /etc/rc.d/rc.local is not marked executable, skipping.
[1044627.933809] traps: bacula-dir[1259686] general protection fault ip:7f9c098a7ee8 sp:7f9c06b97bc8 error:0 in libbac-15.0.3.so[7f9c09880000+66000]
[1131027.300529] traps: bacula-dir[1376697] general protection fault ip:7f46ad45fee8 sp:7f46aa664bc8 error:0 in libbac-15.0.3.so[7f46ad438000+66000]
[1162496.113015] bacula-dir[2407485]: segfault at 555bb4000000 ip 00007f480429b40b sp 00007f47a67fa070 error 4 in libc.so.6[7f4804229000+175000] likely on CPU 2 (core 2, socket 0)
[1162496.131364] Code: f8 64 8b 2b a8 02 75 37 48 8b 15 a8 e9 15 00 64 48 83 3a 00 74 79 48 8d 3d a2 f8 15 00 a8 04 74 0c 48 89 f0 48 25 00 00 00 fc <48> 8b 38 31 d2 e8 db d1 ff ff 64 89 2b 48 83 c4 18 5b 5d c3 90 8b
On Onsdag, December 17, 2025 12:20 CET, "Martin Juhl Prendergast" <m...@rt...> wrote:
> Oh, I can see that..
>
> The storage daemon says:
>
> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
> Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director.
> Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
>
> Config:
>
> Director {
> Name = "bacula-dir"
> Password = "@@SD_PASSWORD@@"
> }
> Director {
> Name = "bacula-mon"
> Password = "@@MON_SD_PASSWORD@@"
> Monitor = yes
> }
> Storage {
> Name = "bacula-sd"
> WorkingDirectory = "/var/spool/bacula"
> PidDirectory = "/var/run"
> PluginDirectory = "/usr/lib64/bacula"
> MaximumConcurrentJobs = 20
> }
> Device {
> Name = "AlwaysIncrement"
> Description = ""
> MediaType = "AlwaysIncrement"
> DeviceType = "File"
> ArchiveDevice = "/home/bacula/autoincrement"
> RemovableMedia = no
> RandomAccess = yes
> AutomaticMount = yes
> LabelMedia = yes
> Autochanger = no
> ReadOnly = no
> MaximumConcurrentJobs = 5
> DriveIndex = 0
> }
> Device {
> Name = "FileChgr1-Dev1"
> MediaType = "File1"
> ArchiveDevice = "/tmp"
> RemovableMedia = no
> RandomAccess = yes
> AutomaticMount = yes
> LabelMedia = yes
> AlwaysOpen = no
> MaximumConcurrentJobs = 5
> }
> Device {
> Name = "FileChgr1-Dev2"
> MediaType = "File1"
> ArchiveDevice = "/tmp"
> RemovableMedia = no
> RandomAccess = yes
> AutomaticMount = yes
> LabelMedia = yes
> AlwaysOpen = no
> MaximumConcurrentJobs = 5
> }
> Device {
> Name = "FileChgr2-Dev1"
> MediaType = "File2"
> ArchiveDevice = "/tmp"
> RemovableMedia = no
> RandomAccess = yes
> AutomaticMount = yes
> LabelMedia = yes
> AlwaysOpen = no
> MaximumConcurrentJobs = 5
> }
> Device {
> Name = "FileChgr2-Dev2"
> MediaType = "File2"
> ArchiveDevice = "/tmp"
> RemovableMedia = no
> RandomAccess = yes
> AutomaticMount = yes
> LabelMedia = yes
> AlwaysOpen = no
> MaximumConcurrentJobs = 5
> }
> Messages {
> Name = "Standard"
> Director = bacula-dir = All
> }
> Autochanger {
> Name = "FileChgr1"
> Device = "FileChgr1-Dev1"
> Device = "FileChgr1-Dev2"
> ChangerDevice = "/dev/null"
> ChangerCommand = ""
> }
> Autochanger {
> Name = "FileChgr2"
> Device = "FileChgr2-Dev1"
> Device = "FileChgr2-Dev2"
> ChangerDevice = "/dev/null"
> ChangerCommand = ""
> }
> Device {
> DeviceType = "File"
> RemovableMedia = no
> AutomaticMount = yes
> LabelMedia = yes
> MaximumConcurrentJobs = 5
> RandomAccess = yes
> Name = "Consolidate"
> Description = ""
> DriveIndex = 0
> ArchiveDevice = "/home/bacula/consolidate"
> MediaType = "Consolidate"
> ReadOnly = no
> Autochanger = no
> }
>
> Please say if I need to provide more configuration
>
> /Martin
>
>
> Martin,
>
> It looks like your message was cut off. It doesn't have any information after "The storage daemon says".
>
> Regards,
> Robert Gerber
> 402-237-8692
> ro...@cr...
>
>
> On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote:
>
> Hi guys
>
> Hope someone can help me..
>
> I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10..
>
> I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup.
>
> On the hosts I get:
> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted.
> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer
> Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103
>
>
>
> The Storage daemon says
>
>
> _______________________________________________
> Bacula-users mailing list
> Bac...@li...
> https://lists.sourceforge.net/lists/listinfo/bacula-users
|
|
From: Martin J. P. <m...@rt...> - 2025-12-17 11:20:31
|
Oh, I can see that..
The storage daemon says:
Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
Dec 16 21:59:08 ************ bacula-sd[822953]: bacula-sd: record_write.c:236-37 Got write_block_to_dev error on device "Consolidate" (/home/bacula/consolidate). Error sending Volume info to Director.
Dec 16 21:59:10 ************ bacula-sd[822953]: bacula-sd: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to client:************:9103
Config:
Director {
Name = "bacula-dir"
Password = "@@SD_PASSWORD@@"
}
Director {
Name = "bacula-mon"
Password = "@@MON_SD_PASSWORD@@"
Monitor = yes
}
Storage {
Name = "bacula-sd"
WorkingDirectory = "/var/spool/bacula"
PidDirectory = "/var/run"
PluginDirectory = "/usr/lib64/bacula"
MaximumConcurrentJobs = 20
}
Device {
Name = "AlwaysIncrement"
Description = ""
MediaType = "AlwaysIncrement"
DeviceType = "File"
ArchiveDevice = "/home/bacula/autoincrement"
RemovableMedia = no
RandomAccess = yes
AutomaticMount = yes
LabelMedia = yes
Autochanger = no
ReadOnly = no
MaximumConcurrentJobs = 5
DriveIndex = 0
}
Device {
Name = "FileChgr1-Dev1"
MediaType = "File1"
ArchiveDevice = "/tmp"
RemovableMedia = no
RandomAccess = yes
AutomaticMount = yes
LabelMedia = yes
AlwaysOpen = no
MaximumConcurrentJobs = 5
}
Device {
Name = "FileChgr1-Dev2"
MediaType = "File1"
ArchiveDevice = "/tmp"
RemovableMedia = no
RandomAccess = yes
AutomaticMount = yes
LabelMedia = yes
AlwaysOpen = no
MaximumConcurrentJobs = 5
}
Device {
Name = "FileChgr2-Dev1"
MediaType = "File2"
ArchiveDevice = "/tmp"
RemovableMedia = no
RandomAccess = yes
AutomaticMount = yes
LabelMedia = yes
AlwaysOpen = no
MaximumConcurrentJobs = 5
}
Device {
Name = "FileChgr2-Dev2"
MediaType = "File2"
ArchiveDevice = "/tmp"
RemovableMedia = no
RandomAccess = yes
AutomaticMount = yes
LabelMedia = yes
AlwaysOpen = no
MaximumConcurrentJobs = 5
}
Messages {
Name = "Standard"
Director = bacula-dir = All
}
Autochanger {
Name = "FileChgr1"
Device = "FileChgr1-Dev1"
Device = "FileChgr1-Dev2"
ChangerDevice = "/dev/null"
ChangerCommand = ""
}
Autochanger {
Name = "FileChgr2"
Device = "FileChgr2-Dev1"
Device = "FileChgr2-Dev2"
ChangerDevice = "/dev/null"
ChangerCommand = ""
}
Device {
DeviceType = "File"
RemovableMedia = no
AutomaticMount = yes
LabelMedia = yes
MaximumConcurrentJobs = 5
RandomAccess = yes
Name = "Consolidate"
Description = ""
DriveIndex = 0
ArchiveDevice = "/home/bacula/consolidate"
MediaType = "Consolidate"
ReadOnly = no
Autochanger = no
}
Please say if I need to provide more configuration
/Martin
Martin,
It looks like your message was cut off. It doesn't have any information after "The storage daemon says".
Regards,
Robert Gerber
402-237-8692
ro...@cr...
On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote:
Hi guys
Hope someone can help me..
I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10..
I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup.
On the hosts I get:
Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted.
Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer
Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103
The Storage daemon says
_______________________________________________
Bacula-users mailing list
Bac...@li...
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
|
From: Rob G. <ro...@cr...> - 2025-12-17 00:14:12
|
Martin, It looks like your message was cut off. It doesn't have any information after "The storage daemon says". Regards, Robert Gerber 402-237-8692 ro...@cr... On Tue, Dec 16, 2025 at 5:56 PM Martin Juhl Prendergast <m...@rt...> wrote: > Hi guys > > Hope someone can help me.. > > I have just switched from BareOS to Bacula (and bacularis).. Currently > running 15.0.3 on RHEL9+RHEL10.. > > I have configured some hosts, and most of the hosts backs up just fine.. > but the biggest of the machines (backup of a couple of hundreds of GB), > fails during backup. > > On the hosts I get: > Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: > sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to > Storage daemon:*****************:9103, but only 49152 accepted. > Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: > sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. > ERR=Connection reset by peer > Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: > sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on > call to Storage daemon:**************:9103 > > > > The Storage daemon says > > > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users > |
|
From: Martin J. P. <m...@rt...> - 2025-12-16 23:55:42
|
Hi guys Hope someone can help me.. I have just switched from BareOS to Bacula (and bacularis).. Currently running 15.0.3 on RHEL9+RHEL10.. I have configured some hosts, and most of the hosts backs up just fine.. but the biggest of the machines (backup of a couple of hundreds of GB), fails during backup. On the hosts I get: Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:395 Wrote 65355 bytes to Storage daemon:*****************:9103, but only 49152 accepted. Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: backup.c:1056-37 Network send error to SD. ERR=Connection reset by peer Dec 16 22:59:10 sullust.outerrim.lan bacula-fd[3113761]: sullust.outerrim.lan: message.c:1841-37 bsock.c:275 Socket has errors=1 on call to Storage daemon:**************:9103 The Storage daemon says |
|
From: Arno L. <al...@it...> - 2025-12-12 06:50:11
|
Hi Gareth, (sent directly instead of to the list...) Am 12.12.2025 um 01:16 schrieb Gareth Evans: > > >> On 2 Dec 2025, at 08:47, Arno Lehmann <al...@it...> wrote: > >> I usually pick some B-Brand, Supermicro-based systems with an appropriate number of disk bays for such workloads. > > Hi Arno, > > I have an interest in this issue at the moment too. Can you give examples of B-Brands please? Especially those with supermicro motherboards and any good alternatives you may care to note? I suspect my own rather local experience will not help you a lot, but I tend to recommend (and purchase for my own use) server systems from https://www.ico.de/ and could probably find something to recommend in France. In Germany, https://www.thomas-krenn.com/ seems to be somewhat popular, too. Other parts of the world -- tricky for me. My own customers and partners tend to have different ideas where to get their hardware. Cheers, Arno > Many thanks > Gareth -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück |