Menu Content

Support

> Forums, FAQs & Paid Support
Welcome, Guest
Username Password: Remember me

Joomsef crawl website missing more URLS
(1 viewing) (1) Guest
Support forum for customers who have purchased JoomSEF 4 (Joomla 1.6/1.7/2.5+ compatible). Archive only, no new post can be added.

NOTE: This category has been locked. If you have purchased paid version, please, use our Support Ticket system instead. If you are using free edition, please see the Community Support section.
  • Page:
  • 1
  • 2

TOPIC: Joomsef crawl website missing more URLS

Joomsef crawl website missing more URLS 10 years, 4 months ago #45200

Hi,

My site Joomla 3.3.1, Joomsef 4.5.2, k2.
1. click clear cache
2. purge sef urls
3. I use crawl website with default setup (root url https//www.exmysite.com/ and level 5),
then Number 53 urls of crawls less than normal value (about 600 urls), This function is normal for a few days ago. I don't have any to change the settings.
how to fix this?

Thanks,
The topic has been locked.

Re: Joomsef crawl website missing more URLS 10 years, 4 months ago #45202

  • dajo
  • OFFLINE
  • Posts: 5069
Hi,

Does the crawler finish successfully or is there an error message displayed?
ARTIO Support Team
The topic has been locked.

Re: Joomsef crawl website missing more URLS 10 years, 4 months ago #45203

dajo wrote:
Hi,

Does the crawler finish successfully or is there an error message displayed?

Hi,

it is success.

.htaccess as following:
##
# @package Joomla
# @copyright Copyright (C) 2005 - 2013 Open Source Matters. All rights reserved.
# @license GNU General Public License version 2 or later; see LICENSE.txt
##

##
# READ THIS COMPLETELY IF YOU CHOOSE TO USE THIS FILE!
#
# The line just below this section: 'Options +FollowSymLinks' may cause problems
# with some server configurations. It is required for use of mod_rewrite, but may already
# be set by your server administrator in a way that dissallows changing it in
# your .htaccess file. If using it causes your server to error out, comment it out (add # to
# beginning of line), reload your site in your browser and test your sef url's. If they work,
# it has been set by your server administrator and you do not need it set here.
##

## Can be commented out if causes errors, see notes above.
#Options +FollowSymLinks

## Mod_rewrite in use.

RewriteEngine On

## Begin - Rewrite rules to block out some common exploits.
# If you experience problems on your site block out the operations listed below
# This attempts to block the most common type of exploit `attempts` to Joomla!
#
# Block out any script trying to base64_encode data within the URL.
RewriteCond %{QUERY_STRING} base64_encode[^(]*\([^)]*\) [OR]
# Block out any script that includes a <script> tag in URL.
RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
# Block out any script trying to set a PHP GLOBALS variable via URL.
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
# Block out any script trying to modify a _REQUEST variable via URL.
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2})
# Return 403 Forbidden header and show the content of the root homepage
RewriteRule .* index.php [F]
#
## End - Rewrite rules to block out some common exploits.

## Begin - Custom redirects
#
# If you need to redirect some pages, or set a canonical non-www to
# www redirect (or vice versa), place that code here. Ensure those
# redirects use the correct RewriteRule syntax and the [R=301,L] flags.
#
## End - Custom redirects

##
# Uncomment following line if your webserver's URL
# is not directly related to physical file paths.
# Update Your Joomla! Directory (just / for root).
##

# RewriteBase //

## Begin - Joomla! core SEF Section.
#
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
#
# If the requested path and file is not /index.php and the request
# has not already been internally rewritten to the index.php script
RewriteCond %{REQUEST_URI} !^/index\.php
# and the request is for something within the component folder,
# or for the site root, or for an extensionless URL, or the
# requested URL ends with one of the listed extensions
RewriteCond %{REQUEST_URI} /component/|(/[^.]*|\.(php|html?|feed|pdf|vcf|raw))$ [NC]
# and the requested path and file doesn't directly match a physical file
RewriteCond %{REQUEST_FILENAME} !-f
# and the requested path and file doesn't directly match a physical folder
RewriteCond %{REQUEST_FILENAME} !-d
# internally rewrite the request to the index.php script
RewriteRule .* index.php [L]
#
## End - Joomla! core SEF Section.
The topic has been locked.

Re: Joomsef crawl website missing more URLS 10 years, 4 months ago #45205

  • dajo
  • OFFLINE
  • Posts: 5069
Hi,

.htaccess file doesn't affect the website crawler.

If it was working recently there must have been some change that caused the crawler not recognizing URLs in your website's source code. It is difficult to say what could cause it, maybe some plugin in Joomla modifies the source code in some way.

What if you manually browse through your webiste? Do more SEF URLs get generated in JoomSEF? It can also be caused by disabling SEF for some components in JoomSEF, because the crawler only follows SEF URLs.
ARTIO Support Team
The topic has been locked.

Re: Joomsef crawl website missing more URLS 10 years, 4 months ago #45222

dajo wrote:
Hi,

.htaccess file doesn't affect the website crawler.

If it was working recently there must have been some change that caused the crawler not recognizing URLs in your website's source code. It is difficult to say what could cause it, maybe some plugin in Joomla modifies the source code in some way.

What if you manually browse through your webiste? Do more SEF URLs get generated in JoomSEF? It can also be caused by disabling SEF for some components in JoomSEF, because the crawler only follows SEF URLs.


Hi,

Now I have fix it, URL working fine.
and,
Maybe I found a bug: recently Joomsef not recognized menu item type-External URL point to k2 categories.

Now I change main menu item type-External URL to K2-categories, then joomsef and crawler URL is normal.

for a long period of time use menu item type-External URL point to k2 categories, joomsef URL work fine, now it like not recognized.

Also Joomsef is good for joomla with k2.

Thanks,




Maybe
The topic has been locked.

Re: Joomsef crawl website missing more URLS 10 years, 4 months ago #45228

  • dajo
  • OFFLINE
  • Posts: 5069
Hi,

Your External URL probably wasn't entered in correct relative format, so it wasn't converted to SEF correctly, thus preventing JoomSEF from crawling the page.
ARTIO Support Team
The topic has been locked.
  • Page:
  • 1
  • 2
User Login Empty