<div dir="ltr"><div>Hi Andreas, <br></div><div>Linux version 3.10.0-693.5.2.el7.x86_64 (<a href="mailto:builder@kbuilder.dev.centos.org" target="_blank">builder@kbuilder.dev.centos.org</a>) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Fri Oct 20 20:</div>32:50 UTC 2017<br><div>Lustre: Lustre: Build Version: 2.10.1</div><div>Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz<br></div><div><br></div><div>It is reading in up to 256 threads. And writing 16 files in up to 16 threads. </div><div><br></div><div>It is reproducible (but does not fail every time) on this particular machine, which might just be a particular network timing.</div><div>I will try to reproduce it on another machine and get back to you if successful. <br></div><div><br></div><div>Any ideas why this lock would have failed?</div><div>A quick analysis shows that the only place where our_vma is called is lustre/llite/vvp_io.c:453, and it only acquires read lock: <br></div><div>vvp_mmap_locks: <br></div><div>452 down_read(&mm->mmap_sem);<br> 453 while((vma = our_vma(mm, addr, count)) != NULL) {<br> 454 struct dentry *de = file_dentry(vma->vm_file);<br> 455 struct inode *inode = de->d_inode;<br> 456 int flags = CEF_MUST;</div><div><br></div><div>whereas our_vma has this: <br></div><div>70 /* mmap_sem must have been held by caller. */<br>71 LASSERT(!down_write_trylock(&mm->mmap_sem));</div><div><br></div><div>So i guess if there are multiple threads in vvp_mmap_locks and more than one happen to acquire read_lock, or one of them acquires write lock then the other would fail, no?</div><div>I will put these details into JIRA.<br></div><div>Jacek Tomaka<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 2, 2019 at 3:45 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The best place to report Lustre bugs is at <a href="https://jira.whamcloud.com/" rel="noreferrer" target="_blank">https://jira.whamcloud.com/</a><br>
<br>
Please include the Lustre version number you are running, and any details you can provide about what kind of IO the Java application was doing at the time, if this is even possible for Java :-). It looks like it is doing AIO? Also, is this repeatable, or a one-time event?<br>
<br>
Cheers, Andreas<br>
<br>
> On Jul 2, 2019, at 01:20, Jacek Tomaka <<a href="mailto:jacekt@dug.com" target="_blank">jacekt@dug.com</a>> wrote:<br>
> <br>
> Hello,<br>
> I was wondering if you would be interested in the following failed assertion:<br>
> <br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: LustreError:<br>
> 251884:0:(llite_mmap.c:71:our_vma()) ASSERTION(<br>
> !down_write_trylock(&mm->mmap_sem) ) failed:<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: LustreError:<br>
> 251884:0:(llite_mmap.c:71:our_vma()) LBUG<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: Pid: 251884, comm: java<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: #012Call Trace:<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc03d67ae>]<br>
> libcfs_call_trace+0x4e/0x60 [libcfs]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc03d683c>]<br>
> lbug_with_loc+0x4c/0xb0 [libcfs]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc116e66b>]<br>
> our_vma+0x16b/0x170 [lustre]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc11857f9>]<br>
> vvp_io_rw_lock+0x409/0x6e0 [lustre]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc0fbb312>] ?<br>
> lov_io_iter_init+0x302/0x8b0 [lov]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc1185b29>]<br>
> vvp_io_write_lock+0x59/0xf0 [lustre]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc063ebec>]<br>
> cl_io_lock+0x5c/0x3d0 [obdclass]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc063f1db>]<br>
> cl_io_loop+0x11b/0xc90 [obdclass]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc1133258>]<br>
> ll_file_io_generic+0x498/0xc40 [lustre]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc1133cdd>]<br>
> ll_file_aio_write+0x12d/0x1f0 [lustre]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffffc1133e6e>]<br>
> ll_file_write+0xce/0x1e0 [lustre]<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffff81200cad>]<br>
> vfs_write+0xbd/0x1e0<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffff8111f394>] ?<br>
> __audit_syscall_entry+0xb4/0x110<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffff81201abf>]<br>
> SyS_write+0x7f/0xe0<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: [<ffffffff816b5292>]<br>
> tracesys+0xdd/0xe2<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel:<br>
> 2019-07-02T01:45:11-05:00 nanny1926 kernel: Kernel panic - not syncing: LBUG<br>
> <br>
> Is there any other place where you would want it reported?<br>
> <br>
> -- <br>
> Jacek Tomaka<br>
> Geophysical Software Developer<br>
> <br>
> <br>
> <br>
> <br>
> DownUnder GeoSolutions<br>
> <br>
> 76 Kings Park Road<br>
> West Perth 6005 WA, Australia<br>
> tel +61 8 9287 4143<br>
> <a href="mailto:jacekt@dug.com" target="_blank">jacekt@dug.com</a><br>
> <a href="http://www.dug.com" rel="noreferrer" target="_blank">www.dug.com</a><br>
> _______________________________________________<br>
> lustre-devel mailing list<br>
> <a href="mailto:lustre-devel@lists.lustre.org" target="_blank">lustre-devel@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_2089838538586004230m_-932549572847940553gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><font color="#000000"><font face="arial,helvetica,sans-serif"><b>Jacek Tomaka</b></font></font><br><font color="#000000"><font face="arial,helvetica,sans-serif"><font size="2">Geophysical Software Developer</font></font></font><br></div><font face="arial,helvetica,sans-serif" color="#000000">
</font><div><span lang="EN-US"></span> <span lang="EN-US"><b><br><br></b></span></div><font face="arial,helvetica,sans-serif" color="#000000">
</font><img src="http://drive.google.com/uc?export=view&id=0B4X9ixpc-ZU_NHV0WnluaXp5ZkE"><br><br><span style="color:rgb(102,102,102)"><font size="2"><b>DownUnder GeoSolutions<br><br></b></font></span><div><span style="color:rgb(102,102,102)"></span><span style="color:rgb(102,102,102)">76 Kings Park Road<br></span></div><span style="color:rgb(102,102,102)">West Perth 6005 WA, Australia<br><i><b>tel </b></i><a href="tel:+61%208%209287%204143" value="+61892874143" target="_blank">+61 8 9287 4143</a><br><a href="mailto:jacekt@dug.com" target="_blank">jacekt@dug.com</a><br><b><a href="http://www.dug.com" target="_blank">www.dug.com</a></b></span></div></div></div></div></div></div></span></div></div></div></div></div></div></div>